[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505148206 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003318] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009006] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013046] *** VALIDATE selinux *** [ 0.021339] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027009] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029079] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030072] *** VALIDATE tmpfs *** [ 0.032405] *** VALIDATE proc *** [ 0.034110] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.036305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043722] debug: unmapping init [mem 0xffffffff87859000-0xffffffff87860fff] [ 0.045183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046677] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047017] ... version: 2 [ 0.048006] ... bit width: 48 [ 0.049005] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051006] ... max period: 00007fffffffffff [ 0.052006] ... fixed-purpose events: 3 [ 0.053005] ... event mask: 000000070000000f [ 0.055286] rcu: Hierarchical SRCU implementation. [ 0.057540] smp: Bringing up secondary CPUs ... [ 0.058476] x86: Booting SMP configuration: [ 0.059012] .... node #0, CPUs: #1 #2 #3 [ 0.065018] smp: Brought up 1 node, 4 CPUs [ 0.067006] smpboot: Max logical packages: 1 [ 0.068007] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235020] node 0 deferred pages initialised in 166ms [ 0.238010] devtmpfs: initialized [ 0.239025] x86/mm: Memory block size: 128MB [ 0.241332] gcov: version magic: 0x41383552 [ 0.244257] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247408] pinctrl core: initialized pinctrl subsystem [ 0.249210] [ 0.250007] ************************************************************* [ 0.252006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253005] ** ** [ 0.255006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257006] ** ** [ 0.258007] ** This means that this kernel is built to expose internal ** [ 0.260023] ** IOMMU data structures, which may compromise security on ** [ 0.262007] ** your system. ** [ 0.264013] ** ** [ 0.266009] ** If you see this message and you are not debugging the ** [ 0.267006] ** kernel, report this immediately to your vendor! ** [ 0.270007] ** ** [ 0.272007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.273005] ************************************************************* [ 0.274546] NET: Registered protocol family 16 [ 0.276320] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.278042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.280035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.283442] cpuidle: using governor menu [ 0.284693] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286498] PCI: Using configuration type 1 for base access [ 0.288140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.298062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.300037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.303121] cryptd: max_cpu_qlen set to 1000 [ 0.306281] ACPI: Added _OSI(Module Device) [ 0.307010] ACPI: Added _OSI(Processor Device) [ 0.309008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.310007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.313601] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.319450] ACPI: Interpreter enabled [ 0.321049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.322009] ACPI: Using IOAPIC for interrupt routing [ 0.323074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.326430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.335720] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.340010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.343055] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.347258] acpiphp: Slot [2] registered [ 0.348126] acpiphp: Slot [3] registered [ 0.349073] acpiphp: Slot [4] registered [ 0.351068] acpiphp: Slot [5] registered [ 0.352091] acpiphp: Slot [6] registered [ 0.353203] acpiphp: Slot [7] registered [ 0.355130] acpiphp: Slot [8] registered [ 0.356067] acpiphp: Slot [9] registered [ 0.357071] acpiphp: Slot [10] registered [ 0.358068] acpiphp: Slot [11] registered [ 0.359064] acpiphp: Slot [12] registered [ 0.361087] acpiphp: Slot [13] registered [ 0.362069] acpiphp: Slot [14] registered [ 0.363062] acpiphp: Slot [15] registered [ 0.365075] acpiphp: Slot [16] registered [ 0.366074] acpiphp: Slot [17] registered [ 0.368100] acpiphp: Slot [18] registered [ 0.369100] acpiphp: Slot [19] registered [ 0.370068] acpiphp: Slot [20] registered [ 0.372061] acpiphp: Slot [21] registered [ 0.373067] acpiphp: Slot [22] registered [ 0.374071] acpiphp: Slot [23] registered [ 0.375060] acpiphp: Slot [24] registered [ 0.377075] acpiphp: Slot [25] registered [ 0.378097] acpiphp: Slot [26] registered [ 0.379065] acpiphp: Slot [27] registered [ 0.380063] acpiphp: Slot [28] registered [ 0.382059] acpiphp: Slot [29] registered [ 0.383058] acpiphp: Slot [30] registered [ 0.384075] acpiphp: Slot [31] registered [ 0.385083] PCI host bridge to bus 0000:00 [ 0.386016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.388016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.390018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.392016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.395015] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.397017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.399177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.403288] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.407530] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.415025] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.420068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.422011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.424018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.426011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428464] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.431853] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.434039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.436712] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.441015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.454017] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.459000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.464741] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.481025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.528010] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.534017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.540015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.559039] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.568620] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.581020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.589016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.609015] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.621320] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.631015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.640014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.658021] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.671956] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.681013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.688012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.709015] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.719000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.725015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.730017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.758017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.766375] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.768307] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.770398] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.771260] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.774165] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.779098] iommu: Default domain type: Passthrough [ 0.780477] SCSI subsystem initialized [ 0.782089] ACPI: bus type USB registered [ 0.783060] usbcore: registered new interface driver usbfs [ 0.784035] usbcore: registered new interface driver hub [ 0.787085] usbcore: registered new device driver usb [ 0.789254] pps_core: LinuxPPS API ver. 1 registered [ 0.791007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.793038] PTP clock support registered [ 0.796052] EDAC MC: Ver: 3.0.0 [ 0.797522] PCI: Using ACPI for IRQ routing [ 0.798801] NetLabel: Initializing [ 0.799000] NetLabel: domain hash size = 128 [ 0.801010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.802061] NetLabel: unlabeled traffic allowed by default [ 0.805044] vgaarb: loaded [ 0.806247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.807009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.812889] clocksource: Switched to clocksource kvm-clock [ 0.919963] VFS: Disk quotas dquot_6.6.0 [ 0.921261] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.923323] *** VALIDATE ramfs *** [ 0.924468] *** VALIDATE hugetlbfs *** [ 0.926279] pnp: PnP ACPI init [ 0.928366] pnp: PnP ACPI: found 6 devices [ 0.945705] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.948376] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.950148] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.951822] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.953840] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.956015] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.958617] NET: Registered protocol family 2 [ 0.961263] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.965526] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.968172] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972539] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.975277] TCP: Hash tables configured (established 65536 bind 65536) [ 0.977761] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.981776] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.985374] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989499] NET: Registered protocol family 1 [ 0.995202] RPC: Registered named UNIX socket transport module. [ 0.996996] RPC: Registered udp transport module. [ 0.998493] RPC: Registered tcp transport module. [ 1.003182] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005162] NET: Registered protocol family 44 [ 1.006427] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.008162] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.014085] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.016680] PCI: CLS 0 bytes, default 64 [ 1.018163] Unpacking initramfs... [ 2.475575] debug: unmapping init [mem 0xffff8c4c3cc54000-0xffff8c4c3ffbffff] [ 2.481045] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.483272] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.486888] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.086250] Initialise system trusted keyrings [ 3.087719] Key type blacklist registered [ 3.089438] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.098619] zbud: loaded [ 3.101257] *** VALIDATE nfs *** [ 3.102475] *** VALIDATE nfs4 *** [ 3.107781] pstore: using deflate compression [ 3.111079] Platform Keyring initialized [ 3.234778] NET: Registered protocol family 38 [ 3.236226] Key type asymmetric registered [ 3.237441] Asymmetric key parser 'x509' registered [ 3.243329] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.246615] io scheduler mq-deadline registered [ 3.248093] io scheduler kyber registered [ 3.249487] io scheduler bfq registered [ 3.250967] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.254066] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.256394] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.258251] ACPI: Power Button [PWRF] [ 3.361183] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.458788] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.662680] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.766485] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.974754] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.003486] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.032557] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.037024] Non-volatile memory driver v1.3 [ 4.038319] Linux agpgart interface v0.103 [ 4.061620] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 4.068742] vda: detected capacity change from 0 to 68521984 [ 4.091418] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.093661] vdb: detected capacity change from 0 to 1073741824 [ 4.114617] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.119230] vdc: detected capacity change from 0 to 2621440000 [ 4.138559] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.141211] vdd: detected capacity change from 0 to 2621440000 [ 4.154049] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.156292] vde: detected capacity change from 0 to 4294967296 [ 4.173805] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.176706] vdf: detected capacity change from 0 to 4294967296 [ 4.184849] libphy: Fixed MDIO Bus: probed [ 4.193000] usbcore: registered new interface driver usbserial_generic [ 4.195390] usbserial: USB Serial support registered for generic [ 4.197617] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.203660] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.205784] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.211809] mousedev: PS/2 mouse device common for all mice [ 4.214242] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.215194] rtc_cmos 00:05: RTC can wake from S4 [ 4.223718] rtc_cmos 00:05: registered as rtc0 [ 4.225358] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.230270] intel_pstate: CPU model not supported [ 4.232664] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.235869] hid: raw HID events driver (C) Jiri Kosina [ 4.237841] usbcore: registered new interface driver usbhid [ 4.240176] usbhid: USB HID core driver [ 4.241818] drop_monitor: Initializing network drop monitor service [ 4.244876] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.244897] Initializing XFRM netlink socket [ 4.252660] NET: Registered protocol family 10 [ 4.255273] Segment Routing with IPv6 [ 4.256444] NET: Registered protocol family 17 [ 4.259497] mpls_gso: MPLS GSO support [ 4.264059] RAS: Correctable Errors collector initialized. [ 4.266096] AVX version of gcm_enc/dec engaged. [ 4.267498] AES CTR mode by8 optimization enabled [ 4.371059] sched_clock: Marking stable (4371029115, 0)->(5131454880, -760425765) [ 4.374431] registered taskstats version 1 [ 4.376338] Loading compiled-in X.509 certificates [ 4.378384] zswap: loaded using pool lzo/zbud [ 4.415925] Key type big_key registered [ 4.433205] Key type encrypted registered [ 4.434738] ima: No TPM chip found, activating TPM-bypass! [ 4.437710] ima: Allocated hash algorithm: sha1 [ 4.442386] ima: No architecture policies found [ 4.444148] evm: Initialising EVM extended attributes: [ 4.446171] evm: security.selinux [ 4.447388] evm: security.ima [ 4.448522] evm: security.capability [ 4.449802] evm: HMAC attrs: 0x1 [ 4.454894] rtc_cmos 00:05: setting system clock to 2025-11-23 05:34:10 UTC (1763876050) [ 4.461899] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 4.466926] debug: unmapping init [mem 0xffffffff87582000-0xffffffff87858fff] [ 4.474143] Write protecting the kernel read-only data: 28672k [ 4.481022] debug: unmapping init [mem 0xffffffff85c03000-0xffffffff85dfffff] [ 4.483801] debug: unmapping init [mem 0xffffffff86514000-0xffffffff865fffff] [ 4.518317] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.526382] systemd[1]: Detected virtualization kvm. [ 4.528175] systemd[1]: Detected architecture x86-64. [ 4.529683] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.555711] systemd[1]: No hostname configured. [ 4.564830] systemd[1]: Set hostname to . [ 4.566749] random: systemd: uninitialized urandom read (16 bytes read) [ 4.568901] systemd[1]: Initializing machine ID from random generator. [ 4.730524] random: systemd: uninitialized urandom read (16 bytes read) [ 4.737501] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.746967] random: systemd: uninitialized urandom read (16 bytes read) [ 4.749446] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.760495] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.531120] device-mapper: uevent: version 1.0.3 [ 5.532885] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.420591] virtio_net virtio0 ens2: renamed from eth0 [ 6.453863] random: fast init done [ 6.729457] scsi host0: ata_piix [ 6.775475] scsi host1: ata_piix [ 6.776836] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.780625] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.965554] random: crng init done [ 10.966955] random: 7 urandom warning(s) missed due to ratelimiting [ 10.974580] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.695051] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.924604] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.222316] SELinux: Disabled at runtime. [ 13.288307] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.296709] systemd[1]: Detected virtualization kvm. [ 13.298162] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.860284] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.863555] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.868621] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.871712] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.873904] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.880982] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.885544] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [[ 14.096823] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 14.496486] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.096774] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.106699] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.324200] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.370294] EDAC sbridge: Ver: 1.1.2 [ 18.106893] Key type dns_resolver registered [ 18.457695] NFS: Registering the id_resolver key type [ 18.459062] Key type id_resolver registered [ 18.460678] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg308-server login: [ 40.830887] libcfs: loading out-of-tree module taints kernel. [ 40.840654] Key type ._llcrypt registered [ 40.841713] Key type .llcrypt registered [ 40.878559] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 48.335265] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 48.951357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.956255] alg: No test for adler32 (adler32-zlib) [ 49.925975] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 50.196353] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 51.807251] Key type lgssc registered [ 52.354174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.387734] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 58.708796] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.353540] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.646596] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.009160] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.994141] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 75.379345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.403261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.410290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.499416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.510800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.554986] Lustre: lustre-MDT0000: new disk, initializing [ 76.579714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.586618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 77.880939] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.105451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.133494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 83.161615] Lustre: 6488:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 83.173711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 83.175293] Lustre: Skipped 1 previous similar message [ 83.210662] Lustre: lustre-MDT0001: new disk, initializing [ 83.230479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 83.237975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 83.240503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 84.562157] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.904433] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 89.964873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.989536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 90.070845] Lustre: lustre-OST0000: new disk, initializing [ 90.072356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 90.088153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.142592] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.741838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 93.744806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 93.772943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 98.049055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.073441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 98.113988] Lustre: lustre-OST0001: new disk, initializing [ 98.116054] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 98.133955] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.032238] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.292042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 101.294574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 101.308459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 105.963821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 112.660162] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 119.536843] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing check_logdir /tmp/testlogs/ [ 120.981345] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing yml_node [ 122.631353] Lustre: DEBUG MARKER: Client: 2.16.61.40 [ 123.506191] Lustre: DEBUG MARKER: MDS: 2.16.61.40 [ 124.368517] Lustre: DEBUG MARKER: OSS: 2.16.61.40 [ 124.886827] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 23 00:36:10 EST 2025 [ 130.853953] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 131.396759] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 144.863697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 144.865838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 144.869252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 147.423461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 147.423906] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.425452] Lustre: Skipped 2 previous similar messages [ 147.428606] Lustre: Skipped 2 previous similar messages [ 149.245446] LustreError: 12191:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 149.272531] Lustre: server umount lustre-MDT0000 complete [ 150.461742] LustreError: 6479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876196 with bad export cookie 3144865271778186046 [ 150.463017] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 150.464855] LustreError: 6479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 152.543897] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.544297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 152.549864] Lustre: Skipped 1 previous similar message [ 155.551582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 155.553577] Lustre: Skipped 2 previous similar messages [ 156.604325] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 156.606176] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 156.645715] Lustre: server umount lustre-MDT0001 complete [ 158.076073] LustreError: 12596:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 158.077942] LustreError: 12596:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 158.095468] Lustre: server umount lustre-OST0000 complete [ 159.351338] Lustre: server umount lustre-OST0001 complete [ 161.377139] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 164.661504] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 167.957210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.109763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.157836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 174.355248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.662669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.686584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.774136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.787731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.827115] Lustre: lustre-MDT0000: new disk, initializing [ 176.856655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.863287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.222142] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 182.343929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.366186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 182.406193] Lustre: 14972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.525103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.870687] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 187.143568] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 188.376880] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 190.603761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.635408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 190.743939] Lustre: lustre-OST0000: new disk, initializing [ 190.745661] Lustre: Skipped 1 previous similar message [ 190.747888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.750059] Lustre: Skipped 2 previous similar messages [ 190.780352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.782930] Lustre: Skipped 1 previous similar message [ 191.870610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.873180] Lustre: Skipped 1 previous similar message [ 191.875813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.909594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.665135] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 195.845991] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 197.246254] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 197.320485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 198.648958] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 198.715891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 202.211365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 202.211777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 202.214805] LustreError: Skipped 1 previous similar message [ 202.220102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 202.223438] Lustre: Skipped 1 previous similar message [ 205.423069] LustreError: 17911:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 205.425039] LustreError: 17911:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 205.463397] Lustre: server umount lustre-OST0000 complete [ 207.328854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.332232] Lustre: Skipped 2 previous similar messages [ 212.448037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 212.450938] Lustre: Skipped 3 previous similar messages [ 221.151115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 221.185934] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 221.189026] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 221.216676] Lustre: server umount lustre-MDT0000 complete [ 222.512267] LustreError: 14965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876268 with bad export cookie 3144865271778187565 [ 222.515692] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 222.516178] LustreError: 14965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 222.642876] Lustre: server umount lustre-MDT0001 complete [ 223.818969] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 00:37:49 (1763876269) [ 226.115364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.267989] LustreError: 18838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 226.274941] LustreError: 18838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 226.292570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.706279] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 230.630747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 232.120217] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 233.958817] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 235.272330] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 237.570977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 239.607087] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 241.470176] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 261.601370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 261.603902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.607584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 261.609830] Lustre: Skipped 2 previous similar messages [ 265.772753] LustreError: 21105:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 265.775415] LustreError: 21105:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 265.812102] Lustre: server umount lustre-OST0000 complete [ 273.259981] Lustre: server umount lustre-MDT0000 complete [ 274.574495] LustreError: 18820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876320 with bad export cookie 3144865271778188447 [ 274.578614] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.703985] Lustre: server umount lustre-MDT0001 complete [ 278.412302] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 279.549660] Key type lgssc unregistered [ 279.688334] LNet: 22040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.690512] LNetError: 22040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.698354] LNet: Removed LNI 192.168.203.108@tcp [ 280.039113] Key type .llcrypt unregistered [ 280.040125] Key type ._llcrypt unregistered [ 289.231062] Key type ._llcrypt registered [ 289.231943] Key type .llcrypt registered [ 289.265228] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 289.681599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 289.689514] alg: No test for adler32 (adler32-zlib) [ 290.540339] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 290.626000] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 292.199141] Key type lgssc registered [ 292.566491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 295.063718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 295.070270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.225644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.471870] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 300.226377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 300.346869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 301.730526] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 303.542840] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 304.760169] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 307.137531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 307.272707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.157627] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 310.973407] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 321.637974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 326.112821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 326.115293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.119184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 327.137035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.138162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 327.138987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 327.534895] LustreError: 26117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 327.569957] Lustre: server umount lustre-OST0000 complete [ 330.947304] LustreError: 26318:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 330.949740] LustreError: 26318:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 330.987333] Lustre: server umount lustre-MDT0000 complete [ 331.231853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 331.234257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.238245] LustreError: 23428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.246472] LustreError: 23428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 332.256936] LustreError: 23428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.267391] LustreError: 23428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 332.275466] LustreError: 23389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876378 with bad export cookie 16901305986783738507 [ 332.278852] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.352327] LustreError: 26520:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 332.354213] LustreError: 26520:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 332.416929] Lustre: server umount lustre-MDT0001 complete [ 336.105627] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 337.100948] Key type lgssc unregistered [ 337.232332] LNet: 27053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.235164] LNetError: 27053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 337.242290] LNet: Removed LNI 192.168.203.108@tcp [ 337.524424] Key type .llcrypt unregistered [ 337.525787] Key type ._llcrypt unregistered [ 346.731275] Key type ._llcrypt registered [ 346.732267] Key type .llcrypt registered [ 346.767461] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 347.138623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 347.205049] alg: No test for adler32 (adler32-zlib) [ 348.078088] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 348.155621] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 349.735145] Key type lgssc registered [ 350.095550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 352.743615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 352.749339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 353.910865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.231455] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 358.144400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 358.283988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 359.690429] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 361.547877] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 362.750884] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 367.247905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 367.378994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.346471] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 371.140752] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 378.662860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 383.970357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 383.971030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.977303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 383.978979] Lustre: Skipped 1 previous similar message [ 384.877223] LustreError: 31034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 384.910954] Lustre: server umount lustre-OST0000 complete [ 388.220714] LustreError: 31235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 388.223193] LustreError: 31235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 388.247561] Lustre: server umount lustre-MDT0000 complete [ 389.089915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.090792] LustreError: 28424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.094568] Lustre: Skipped 2 previous similar messages [ 389.100326] LustreError: 28424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 389.104666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 389.106688] LustreError: Skipped 1 previous similar message [ 389.517194] LustreError: 31237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876435 with bad export cookie 16875316261218521606 [ 389.520020] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 389.520023] LustreError: 31237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 389.581945] LustreError: 31438:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 389.583812] LustreError: 31438:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 389.631598] Lustre: server umount lustre-MDT0001 complete [ 393.284321] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 394.360914] Key type lgssc unregistered [ 394.499410] LNet: 31970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.503204] LNetError: 31970:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 394.513358] LNet: Removed LNI 192.168.203.108@tcp [ 394.814903] Key type .llcrypt unregistered [ 394.816593] Key type ._llcrypt unregistered [ 404.052073] Key type ._llcrypt registered [ 404.053230] Key type .llcrypt registered [ 404.102855] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 404.461192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 404.503973] alg: No test for adler32 (adler32-zlib) [ 405.375244] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 405.477637] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 407.063181] Key type lgssc registered [ 407.484938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 410.305208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 410.314607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.486980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.901825] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 415.878456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 415.999642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.353146] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 419.277253] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 420.520791] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 422.809184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 422.925334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.876174] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 426.759200] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 436.259736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 441.312345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 441.314665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 441.316278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.319691] Lustre: Skipped 1 previous similar message [ 442.094113] LustreError: 35607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 442.124707] Lustre: server umount lustre-OST0000 complete [ 445.501876] LustreError: 35807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 445.504691] LustreError: 35807:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 445.541756] Lustre: server umount lustre-MDT0000 complete [ 446.432373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.436342] LustreError: 33339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.440087] LustreError: 33339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 446.809340] LustreError: 35808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876492 with bad export cookie 8584523142765905541 [ 446.812181] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.813786] LustreError: 35808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 446.883837] LustreError: 36009:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 446.886825] LustreError: 36009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 446.936710] Lustre: server umount lustre-MDT0001 complete [ 450.357990] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 451.316957] Key type lgssc unregistered [ 451.446349] LNet: 36539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.449041] LNetError: 36539:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 451.459289] LNet: Removed LNI 192.168.203.108@tcp [ 451.726247] Key type .llcrypt unregistered [ 451.727748] Key type ._llcrypt unregistered [ 459.825132] Key type ._llcrypt registered [ 459.826030] Key type .llcrypt registered [ 459.859914] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 460.185065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 460.221370] alg: No test for adler32 (adler32-zlib) [ 461.089623] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 461.168492] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 462.743099] Key type lgssc registered [ 463.090319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 465.490759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 465.499160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.652332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.941060] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 470.570093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 470.680066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.897285] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 473.676851] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 474.885935] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 477.069778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.173629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.023383] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 480.231465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 480.815983] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.346415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.346697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.347193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.348985] LustreError: Skipped 1 previous similar message [ 485.352045] Lustre: Skipped 1 previous similar message [ 486.990466] Lustre: lustre-OST0000: Not available for connect from 192.168.203.8@tcp (stopping) [ 486.992396] Lustre: Skipped 1 previous similar message [ 490.465213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.467333] Lustre: Skipped 1 previous similar message [ 495.585107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.587693] Lustre: Skipped 2 previous similar messages [ 498.143075] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 498.160919] LustreError: 40157:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 498.198071] Lustre: server umount lustre-OST0000 complete [ 511.198085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.199291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 511.202138] Lustre: Skipped 1 previous similar message [ 511.204848] Lustre: Skipped 1 previous similar message [ 517.435401] LustreError: 40534:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 517.437728] LustreError: 40534:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 517.465970] Lustre: server umount lustre-MDT0000 complete [ 518.723202] LustreError: 37888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876564 with bad export cookie 12123147610660671713 [ 518.727265] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.727640] LustreError: 37888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 524.867317] LustreError: 40736:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 524.869154] LustreError: 40736:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 524.927288] Lustre: server umount lustre-MDT0001 complete [ 528.508609] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 529.585458] Key type lgssc unregistered [ 529.712363] LNet: 41267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.714804] LNetError: 41267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 529.722363] LNet: Removed LNI 192.168.203.108@tcp [ 530.046832] Key type .llcrypt unregistered [ 530.047944] Key type ._llcrypt unregistered [ 539.020749] Key type ._llcrypt registered [ 539.021865] Key type .llcrypt registered [ 539.058260] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 539.400443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 539.432325] alg: No test for adler32 (adler32-zlib) [ 540.278399] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 540.360264] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 541.935218] Key type lgssc registered [ 542.301949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 544.842186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 544.850528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.000348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.322341] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 550.163035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 550.295532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 551.665548] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 553.496181] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 554.733553] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 556.985841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 557.109437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.078422] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 560.916149] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 571.430958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 575.967987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.969787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.973061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.480449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 576.482391] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.488633] Lustre: Skipped 1 previous similar message [ 579.986050] LustreError: 44905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 580.021776] Lustre: server umount lustre-MDT0000 complete [ 581.377114] LustreError: 42619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876627 with bad export cookie 18397830327454719246 [ 581.380688] LustreError: 42619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 581.381163] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.599975] LustreError: 44795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.600301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.608251] LustreError: 44795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 581.614637] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 581.617830] Lustre: Skipped 3 previous similar messages [ 586.719968] LustreError: 42633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.720430] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 586.724936] LustreError: 42633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 586.726610] Lustre: Skipped 1 previous similar message [ 587.515419] LustreError: 45106:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 587.517844] LustreError: 45106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 587.563462] Lustre: server umount lustre-MDT0001 complete [ 605.421781] LustreError: 45329:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 605.423450] LustreError: 45329:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 605.464699] Lustre: server umount lustre-OST0000 complete [ 610.987648] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 612.058277] Key type lgssc unregistered [ 612.187456] LNet: 46166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.190219] LNetError: 46166:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.202332] LNet: Removed LNI 192.168.203.108@tcp [ 612.517458] Key type .llcrypt unregistered [ 612.518445] Key type ._llcrypt unregistered [ 622.569272] Key type ._llcrypt registered [ 622.570176] Key type .llcrypt registered [ 622.613585] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 622.988933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 623.051132] alg: No test for adler32 (adler32-zlib) [ 623.903109] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 623.987630] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 625.575140] Key type lgssc registered [ 625.917096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.453188] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 628.459397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 643.871356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 648.991154] LustreError: 47508:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 675.615166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.589444] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 679.452185] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 845.228422] LustreError: 48218:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 845.264618] Lustre: server umount lustre-OST0000 complete [ 850.748305] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 851.877137] Key type lgssc unregistered [ 852.012463] LNet: 49053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.016150] LNetError: 49053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.024394] LNet: Removed LNI 192.168.203.108@tcp [ 852.340310] Key type .llcrypt unregistered [ 852.341711] Key type ._llcrypt unregistered [ 861.386805] Key type ._llcrypt registered [ 861.387756] Key type .llcrypt registered [ 861.428509] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 861.734258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.742265] alg: No test for adler32 (adler32-zlib) [ 862.602651] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 862.694739] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 864.279162] Key type lgssc registered [ 864.675732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.336900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.344224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.494299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.810126] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 872.616788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.743456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.084949] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 875.894242] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.166245] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 879.371850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 879.502157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.382349] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 883.222252] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 887.779792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.781182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.790090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 890.094085] LustreError: 52671:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 890.131872] Lustre: server umount lustre-OST0000 complete [ 892.896816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.897547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 892.900217] Lustre: Skipped 1 previous similar message [ 897.475872] LustreError: 52873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 897.478390] LustreError: 52873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 897.513007] Lustre: server umount lustre-MDT0000 complete [ 898.016482] LustreError: 51792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.022182] LustreError: 51792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 898.810596] LustreError: 50413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763876944 with bad export cookie 6125971928351010902 [ 898.815102] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 898.815594] LustreError: 50413:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 898.881037] LustreError: 53074:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 898.883034] LustreError: 53074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 898.939698] Lustre: server umount lustre-MDT0001 complete [ 902.395128] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 903.439273] Key type lgssc unregistered [ 903.574391] LNet: 53608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.577049] LNetError: 53608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.583249] LNet: Removed LNI 192.168.203.108@tcp [ 903.876085] Key type .llcrypt unregistered [ 903.877125] Key type ._llcrypt unregistered [ 912.701967] Key type ._llcrypt registered [ 912.702971] Key type .llcrypt registered [ 912.740131] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 913.037107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 913.059197] alg: No test for adler32 (adler32-zlib) [ 913.918179] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 913.996190] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 915.583198] Key type lgssc registered [ 915.978383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 918.656038] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 918.662046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 934.111283] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 939.231284] LustreError: 54950:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 965.855230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.707759] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 969.394736] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 971.762825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.959157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.981499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 973.221019] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 975.987890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 976.108354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.421856] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 979.151628] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.452073] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 981.472188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.474305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.477817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.495763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 982.498299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 982.502358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.214213] LustreError: 57227:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 983.266967] Lustre: server umount lustre-OST0000 complete [ 997.612071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.616479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.673273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.677093] Lustre: Skipped 2 previous similar messages [ 1003.771896] LustreError: 57609:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1003.773708] LustreError: 57609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1003.797683] Lustre: server umount lustre-MDT0000 complete [ 1005.026549] LustreError: 56585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877051 with bad export cookie 15248043165146153085 [ 1005.030668] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.359110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1019.394285] LustreError: 57810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1019.396046] LustreError: 57810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1019.444744] Lustre: server umount lustre-MDT0001 complete [ 1023.642498] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1024.813228] Key type lgssc unregistered [ 1024.975379] LNet: 58342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.977454] LNetError: 58342:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.988369] LNet: Removed LNI 192.168.203.108@tcp [ 1025.288453] Key type .llcrypt unregistered [ 1025.289518] Key type ._llcrypt unregistered [ 1034.508290] Key type ._llcrypt registered [ 1034.509589] Key type .llcrypt registered [ 1034.557763] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1035.086472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.099283] alg: No test for adler32 (adler32-zlib) [ 1035.953439] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1036.048259] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1037.631147] Key type lgssc registered [ 1038.013055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.672294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.678354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.855485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.222927] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1046.682837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1046.842702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.304406] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1050.546225] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1051.853920] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1054.106847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1054.245995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.372737] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1058.249770] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1066.531252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1071.589462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.589638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.593235] LustreError: Skipped 1 previous similar message [ 1071.600144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.606058] Lustre: Skipped 1 previous similar message [ 1072.814377] LustreError: 61981:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1072.853413] Lustre: server umount lustre-OST0000 complete [ 1076.706517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.711405] Lustre: Skipped 1 previous similar message [ 1076.713621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.715338] Lustre: Skipped 1 previous similar message [ 1080.264614] LustreError: 62182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1080.267689] LustreError: 62182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1080.314946] Lustre: server umount lustre-MDT0000 complete [ 1081.824375] LustreError: 59714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.828549] LustreError: 59714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1081.841701] LustreError: 61332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877127 with bad export cookie 1273382109426723717 [ 1081.846494] LustreError: 61332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1081.846504] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1081.916457] LustreError: 62384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1081.918262] LustreError: 62384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1081.975621] Lustre: server umount lustre-MDT0001 complete [ 1085.756955] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1086.907352] Key type lgssc unregistered [ 1087.047430] LNet: 62915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.049749] LNetError: 62915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.059622] LNet: Removed LNI 192.168.203.108@tcp [ 1087.474174] Key type .llcrypt unregistered [ 1087.477151] Key type ._llcrypt unregistered [ 1100.247585] Key type ._llcrypt registered [ 1100.249098] Key type .llcrypt registered [ 1100.286274] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1100.754148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.764442] alg: No test for adler32 (adler32-zlib) [ 1101.645604] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1101.758816] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1103.375257] Key type lgssc registered [ 1103.997175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1107.249169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1107.256194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1108.431104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.692879] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1112.578963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1112.701660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1114.085183] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1115.942767] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1117.220483] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1119.638471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1119.777347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.823839] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1123.686089] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1134.124360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1136.375922] Lustre: Failing over lustre-MDT0000 [ 1136.609567] LustreError: 66845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1136.643443] Lustre: server umount lustre-MDT0000 complete [ 1138.656674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1138.661810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.670861] LustreError: 64576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.679924] LustreError: 64576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1139.171525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.173256] LustreError: 64572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.182939] Lustre: Skipped 1 previous similar message [ 1139.191454] LustreError: 64572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1140.362848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.427686] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.503503] Lustre: *** cfs_fail_loc=135, val=0*** [ 1140.554110] LustreError: 67355:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1140.556160] LustreError: 67355:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -2 [ 1140.558517] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1140.558517] [ 1140.561048] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1140.564518] LustreError: 67343:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1140.566488] LustreError: 67343:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1140.568874] LustreError: 67343:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1140.576855] Lustre: server umount lustre-MDT0000 complete [ 1140.578030] LustreError: 67343:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1143.441364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1143.498356] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.623276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.659923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1144.389320] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1145.240294] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1148.907811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1148.920916] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1148.945673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1153.504330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.506285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.509515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.019362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1156.012131] LustreError: 68161:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1156.013771] LustreError: 68161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1156.048788] Lustre: server umount lustre-OST0000 complete [ 1159.138371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1159.139578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.140395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.140402] Lustre: Skipped 1 previous similar message [ 1159.143505] LustreError: Skipped 1 previous similar message [ 1159.148727] Lustre: Skipped 2 previous similar messages [ 1163.903430] LustreError: 68362:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1163.906861] LustreError: 68362:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1163.937568] Lustre: server umount lustre-MDT0000 complete [ 1164.257469] LustreError: 65289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.267031] LustreError: 65289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1165.267374] LustreError: 66196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877211 with bad export cookie 6316726368556342257 [ 1165.270018] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.270392] LustreError: 66196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1165.388152] Lustre: server umount lustre-MDT0001 complete [ 1169.290009] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1170.417820] Key type lgssc unregistered [ 1170.543364] LNet: 69094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.545433] LNetError: 69094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.555303] LNet: Removed LNI 192.168.203.108@tcp [ 1170.869056] Key type .llcrypt unregistered [ 1170.870119] Key type ._llcrypt unregistered [ 1179.973317] Key type ._llcrypt registered [ 1179.974286] Key type .llcrypt registered [ 1180.012109] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1180.319923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.334227] alg: No test for adler32 (adler32-zlib) [ 1181.195237] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1181.278735] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1182.855166] Key type lgssc registered [ 1183.218448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1185.953136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.961073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.132940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.494465] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1191.399271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1191.568356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1192.931927] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1194.800641] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1196.089859] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1198.386541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1198.555124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.630717] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1202.352698] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1213.926280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1220.223400] Lustre: Failing over lustre-MDT0000 [ 1220.356336] LustreError: 72746:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1220.392711] Lustre: server umount lustre-MDT0000 complete [ 1222.623901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1222.627122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.632787] LustreError: 70478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.639252] LustreError: 70478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1223.243077] LustreError: 70474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.460079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.495731] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.554674] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1223.608994] LustreError: 73254:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1223.610787] LustreError: 73254:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -9 [ 1223.612922] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1223.612922] [ 1223.615319] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1223.618380] LustreError: 73242:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1223.620418] LustreError: 73242:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1223.623045] LustreError: 73242:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1223.634744] Lustre: server umount lustre-MDT0000 complete [ 1223.635878] LustreError: 73242:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1224.161663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.167261] Lustre: Skipped 1 previous similar message [ 1225.775223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.811332] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1225.903738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.920623] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1227.276641] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1228.356149] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1231.332891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1231.339497] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1231.357098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1232.351912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1232.356631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.364093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.449865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1236.451255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1236.454893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.381073] LustreError: 74063:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1238.383555] LustreError: 74063:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1238.420146] Lustre: server umount lustre-OST0000 complete [ 1241.570837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.571053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1241.572129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.572135] Lustre: Skipped 1 previous similar message [ 1241.578124] Lustre: Skipped 1 previous similar message [ 1246.688290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.691018] Lustre: Skipped 2 previous similar messages [ 1251.808564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.810195] Lustre: Skipped 1 previous similar message [ 1254.367221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1254.421889] LustreError: 74264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1254.426027] LustreError: 74264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1254.462818] Lustre: server umount lustre-MDT0000 complete [ 1255.876200] LustreError: 72097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877301 with bad export cookie 3222394076699633857 [ 1255.879671] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.881737] LustreError: 72097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1256.019742] Lustre: server umount lustre-MDT0001 complete [ 1260.319052] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1261.762150] Key type lgssc unregistered [ 1261.919620] LNet: 74999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.922832] LNetError: 74999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.929698] LNet: Removed LNI 192.168.203.108@tcp [ 1262.330693] Key type .llcrypt unregistered [ 1262.332584] Key type ._llcrypt unregistered [ 1266.908872] Key type ._llcrypt registered [ 1266.909920] Key type .llcrypt registered [ 1266.956239] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1267.102421] Key type .llcrypt unregistered [ 1267.104627] Key type ._llcrypt unregistered [ 1276.304334] Key type ._llcrypt registered [ 1276.305619] Key type .llcrypt registered [ 1276.345224] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1276.812769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1276.842945] alg: No test for adler32 (adler32-zlib) [ 1277.712886] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1277.798797] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1279.375159] Key type lgssc registered [ 1279.692716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1282.373972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1282.384478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.570655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.101935] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1288.279203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1288.473952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1289.848780] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1291.670687] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1292.944611] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1295.877244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1296.001212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.040813] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1300.045433] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1308.331869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1318.883669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1318.884423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.885683] LustreError: Skipped 1 previous similar message [ 1318.886445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.891514] Lustre: Skipped 1 previous similar message [ 1320.173218] LustreError: 79105:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1320.204557] Lustre: server umount lustre-OST0000 complete [ 1324.000922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.005215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1324.006961] Lustre: Skipped 2 previous similar messages [ 1327.699731] LustreError: 79308:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1327.703289] LustreError: 79308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1327.736830] Lustre: server umount lustre-MDT0000 complete [ 1329.008408] LustreError: 76816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877375 with bad export cookie 11252993711840958805 [ 1329.011227] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1329.012610] LustreError: 76816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1329.087850] LustreError: 79510:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1329.089707] LustreError: 79510:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1329.137163] Lustre: server umount lustre-MDT0001 complete [ 1333.120251] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1334.085243] Key type lgssc unregistered [ 1334.204354] LNet: 80042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.206371] LNetError: 80042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.216327] LNet: Removed LNI 192.168.203.108@tcp [ 1334.538996] Key type .llcrypt unregistered [ 1334.539938] Key type ._llcrypt unregistered [ 1344.079064] Key type ._llcrypt registered [ 1344.080568] Key type .llcrypt registered [ 1344.127969] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1344.645776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.658331] alg: No test for adler32 (adler32-zlib) [ 1345.522138] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1345.608043] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1347.199186] Key type lgssc registered [ 1347.626870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.521192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.529334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.711594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.341284] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1356.246866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1356.379989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.777153] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1359.549880] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1360.687563] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1362.935715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1363.075485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.984263] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1366.776172] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1377.449533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1386.976721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1386.980064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.985494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.001369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1388.002534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.002812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.963179] LustreError: 83679:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1391.017866] Lustre: server umount lustre-OST0000 complete [ 1393.123770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.137699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.143196] Lustre: Skipped 2 previous similar messages [ 1398.240670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1398.242363] Lustre: Skipped 1 previous similar message [ 1398.849959] LustreError: 83882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1398.851890] LustreError: 83882:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1398.880240] Lustre: server umount lustre-MDT0000 complete [ 1400.090597] LustreError: 83580:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877446 with bad export cookie 1399013505593871970 [ 1400.094266] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1400.095688] LustreError: 83580:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1400.162313] LustreError: 84083:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1400.164823] LustreError: 84083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1400.214781] Lustre: server umount lustre-MDT0001 complete [ 1403.711318] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1404.777564] Key type lgssc unregistered [ 1404.905490] LNet: 84615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.907297] LNetError: 84615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.916264] LNet: Removed LNI 192.168.203.108@tcp [ 1405.201852] Key type .llcrypt unregistered [ 1405.202898] Key type ._llcrypt unregistered [ 1414.641300] Key type ._llcrypt registered [ 1414.642548] Key type .llcrypt registered [ 1414.689351] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1415.081345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1415.088666] alg: No test for adler32 (adler32-zlib) [ 1415.963291] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1416.067393] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1417.655164] Key type lgssc registered [ 1418.075703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1420.867225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1420.873532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.039610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.468313] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1426.249715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1426.388608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.693587] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1429.367513] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.505082] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1432.922231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1433.066907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.051387] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1437.042975] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1447.408351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1451.487888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1451.490087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.493694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.513476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.517039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.517444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.523020] Lustre: Skipped 1 previous similar message [ 1456.755229] LustreError: 88252:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1456.789507] Lustre: server umount lustre-OST0000 complete [ 1462.240510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1462.245087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.253135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1464.126097] LustreError: 88454:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1464.127900] LustreError: 88454:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1464.157399] Lustre: server umount lustre-MDT0000 complete [ 1465.444237] LustreError: 87601:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877511 with bad export cookie 5563871710702347789 [ 1465.449813] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.519589] LustreError: 88656:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1465.521242] LustreError: 88656:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1465.575274] Lustre: server umount lustre-MDT0001 complete [ 1469.116157] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1470.174332] Key type lgssc unregistered [ 1470.310352] LNet: 89187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.313282] LNetError: 89187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.324411] LNet: Removed LNI 192.168.203.108@tcp [ 1470.621918] Key type .llcrypt unregistered [ 1470.623058] Key type ._llcrypt unregistered [ 1479.349606] Key type ._llcrypt registered [ 1479.351200] Key type .llcrypt registered [ 1479.394103] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1479.814394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1479.829533] alg: No test for adler32 (adler32-zlib) [ 1480.695519] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1480.791208] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1482.375125] Key type lgssc registered [ 1482.739534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1485.277791] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1485.283198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1500.703298] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1505.823157] LustreError: 90529:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 1532.447218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.345684] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1536.242265] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1538.123046] LustreError: 91384:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1538.149369] Lustre: server umount lustre-OST0000 complete [ 1540.922545] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 00:59:46 (1763877586) [ 1543.105287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.262530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.605525] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1547.399878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1547.525655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1548.858545] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1550.628276] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1551.786317] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1554.002513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1554.135690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1556.073549] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1557.752795] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1568.486055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1577.951849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1577.953902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.957530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1578.977190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1578.977982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1578.978155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1578.989716] Lustre: Skipped 1 previous similar message [ 1582.893242] LustreError: 94479:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1582.894983] LustreError: 94479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1582.928958] Lustre: server umount lustre-OST0000 complete [ 1588.703939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1588.706706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.711604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1590.331945] LustreError: 94683:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1590.334048] LustreError: 94683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1590.363554] Lustre: server umount lustre-MDT0000 complete [ 1591.654576] LustreError: 94684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877637 with bad export cookie 3772494931220796470 [ 1591.658833] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.790909] Lustre: server umount lustre-MDT0001 complete [ 1595.370993] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1596.431557] Key type lgssc unregistered [ 1596.555311] LNet: 95416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.557406] LNetError: 95416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.566347] LNet: Removed LNI 192.168.203.108@tcp [ 1596.859443] Key type .llcrypt unregistered [ 1596.860398] Key type ._llcrypt unregistered [ 1605.698694] Key type ._llcrypt registered [ 1605.699708] Key type .llcrypt registered [ 1605.735950] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1606.076407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1606.136838] alg: No test for adler32 (adler32-zlib) [ 1607.002084] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1607.100527] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1608.687185] Key type lgssc registered [ 1609.112230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1611.871067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1611.876713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.041333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.409510] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1617.837437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1618.032856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1619.530364] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1621.477297] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1622.701556] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1625.411762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1625.597108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.664387] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1629.449238] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1638.888258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1644.001746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1644.001860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1644.008198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1644.009563] Lustre: Skipped 1 previous similar message [ 1648.748082] LustreError: 99068:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1648.783833] Lustre: server umount lustre-OST0000 complete [ 1653.727907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1653.730178] LustreError: Skipped 1 previous similar message [ 1653.731584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.735023] Lustre: Skipped 1 previous similar message [ 1653.737010] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.125731] LustreError: 99271:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1656.127526] LustreError: 99271:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1656.151921] Lustre: server umount lustre-MDT0000 complete [ 1657.409574] LustreError: 96767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877703 with bad export cookie 6198251802061845592 [ 1657.414026] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.487816] LustreError: 99472:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1657.491016] LustreError: 99472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1657.549608] Lustre: server umount lustre-MDT0001 complete [ 1661.159112] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1662.206963] Key type lgssc unregistered [ 1662.337718] LNet: 100003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.341322] LNetError: 100003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.354619] LNet: Removed LNI 192.168.203.108@tcp [ 1662.694467] Key type .llcrypt unregistered [ 1662.695368] Key type ._llcrypt unregistered [ 1670.847427] Key type ._llcrypt registered [ 1670.848344] Key type .llcrypt registered [ 1670.895863] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1671.274112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1671.318215] alg: No test for adler32 (adler32-zlib) [ 1672.163307] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1672.248076] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1673.831141] Key type lgssc registered [ 1674.182993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1676.892958] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1676.898467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1692.319469] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1697.439260] LustreError: 101209:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 1724.063216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.191390] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1728.243620] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.500951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.601052] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1730.605260] LustreError: 102121:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1730.608553] LustreError: 102121:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1730.611444] LustreError: 102121:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1730.642096] Lustre: server umount lustre-MDT0000 complete [ 1730.644514] LustreError: 102121:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1743.327556] LustreError: 102552:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1743.362266] Lustre: server umount lustre-OST0000 complete [ 1746.194624] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1748.972419] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1752.060606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1754.108576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1756.229838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1758.306292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1760.517174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1760.541562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.639585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1760.650210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1760.688067] Lustre: lustre-MDT0000: new disk, initializing [ 1760.711295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.718919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1762.047689] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1766.212675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1766.232391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1766.270056] Lustre: 104884:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1766.294517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1766.297625] Lustre: Skipped 1 previous similar message [ 1766.337876] Lustre: lustre-MDT0001: new disk, initializing [ 1766.372358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1766.385734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1766.389030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1767.630353] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1770.684506] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.823267] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.985567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1774.010565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1774.125416] Lustre: lustre-OST0000: new disk, initializing [ 1774.128454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1774.157786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.992714] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1779.000748] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1779.703770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1779.706731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1779.740013] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1780.371826] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1780.440820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1781.795149] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1781.864488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1784.801785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1784.803317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.805237] LustreError: Skipped 1 previous similar message [ 1784.811635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.814371] Lustre: Skipped 1 previous similar message [ 1788.589113] LustreError: 107821:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1788.590712] LustreError: 107821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1788.630140] Lustre: server umount lustre-OST0000 complete [ 1789.920266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.920990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.923197] Lustre: Skipped 2 previous similar messages [ 1789.925657] Lustre: Skipped 1 previous similar message [ 1795.040112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.041724] Lustre: Skipped 1 previous similar message [ 1800.160050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.161555] Lustre: Skipped 1 previous similar message [ 1804.255126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1804.256692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.260297] Lustre: Skipped 1 previous similar message [ 1804.291127] LustreError: 108022:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1804.293446] LustreError: 108022:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1804.318770] Lustre: server umount lustre-MDT0000 complete [ 1805.524093] LustreError: 108024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877851 with bad export cookie 4458683324899411491 [ 1805.527536] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.528778] LustreError: 108024:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1805.644466] Lustre: server umount lustre-MDT0001 complete [ 1808.361667] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:04:13 (1763877853) [ 1809.468408] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1814.705892] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1817.659859] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1820.992265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1823.198056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1825.234970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1827.322635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1829.680046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1829.705736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.803589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1829.814499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1829.853539] Lustre: lustre-MDT0000: new disk, initializing [ 1829.881513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.889843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1831.273600] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1835.480244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1835.508835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1835.546771] Lustre: 111354:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1835.574272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1835.577027] Lustre: Skipped 1 previous similar message [ 1835.619249] Lustre: lustre-MDT0001: new disk, initializing [ 1835.664594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1835.669555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1837.108181] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1840.237390] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1841.464646] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1843.765296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1843.787807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1843.889775] Lustre: lustre-OST0000: new disk, initializing [ 1843.892739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1843.922083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.925180] Lustre: Skipped 1 previous similar message [ 1844.797206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1844.802192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1844.845736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1845.949730] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1849.100641] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.485156] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1850.566754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1851.922081] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1851.998679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1854.943957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.947706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.953785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.734090] LustreError: 114290:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1858.735713] LustreError: 114290:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1858.771518] Lustre: server umount lustre-OST0000 complete [ 1860.576657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1860.579763] Lustre: Skipped 2 previous similar messages [ 1866.112509] LustreError: 114491:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1866.114415] LustreError: 114491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1866.143611] Lustre: server umount lustre-MDT0000 complete [ 1867.392321] LustreError: 111346:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877913 with bad export cookie 4458683324899412373 [ 1867.395677] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1867.397175] LustreError: 111346:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1867.510202] Lustre: server umount lustre-MDT0001 complete [ 1870.335563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.527369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.937535] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1874.866353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1876.373946] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1878.204765] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1879.387888] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1881.708333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1883.684513] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1885.470527] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1903.471693] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1906.659327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.659879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1906.660455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.660461] Lustre: Skipped 6 previous similar messages [ 1906.665035] Lustre: Skipped 1 previous similar message [ 1906.668901] LustreError: Skipped 1 previous similar message [ 1910.445960] LustreError: 117534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1910.447667] LustreError: 117534:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1910.483867] Lustre: server umount lustre-OST0000 complete [ 1926.111150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1926.170720] Lustre: server umount lustre-MDT0000 complete [ 1927.376141] LustreError: 115149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763877973 with bad export cookie 4458683324899413241 [ 1927.379134] LustreError: 115149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1927.379292] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.450756] LustreError: 117940:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1927.452472] LustreError: 117940:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1927.500232] Lustre: server umount lustre-MDT0001 complete [ 1931.065362] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 1932.036798] Key type lgssc unregistered [ 1932.163395] LNet: 118471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.166940] LNetError: 118471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.174361] LNet: Removed LNI 192.168.203.108@tcp [ 1932.442527] Key type .llcrypt unregistered [ 1932.443438] Key type ._llcrypt unregistered [ 1938.179931] Key type ._llcrypt registered [ 1938.181093] Key type .llcrypt registered [ 1938.219812] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 1944.223184] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 1944.669376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.677087] alg: No test for adler32 (adler32-zlib) [ 1945.541305] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 1945.624879] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 1947.215128] Key type lgssc registered [ 1947.562975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1950.442719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1952.578169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1954.561044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1956.672599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1958.931593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1958.956994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1958.961979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.050241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1960.060979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1960.097358] Lustre: lustre-MDT0000: new disk, initializing [ 1960.122339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1960.129386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1961.419699] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1965.566729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1965.590294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1965.620943] Lustre: 121569:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1965.642328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1965.644026] Lustre: Skipped 1 previous similar message [ 1965.684485] Lustre: lustre-MDT0001: new disk, initializing [ 1965.717200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1965.729198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1965.731911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1967.013750] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1970.096083] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1971.275809] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1973.573214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1973.601677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1973.721300] Lustre: lustre-OST0000: new disk, initializing [ 1973.725150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1973.729113] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1973.791239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.742049] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 1978.857437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1978.860532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1978.872606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1978.937792] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1980.324308] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1980.398132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1981.754965] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1981.844230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1983.971451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.972197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.974755] LustreError: Skipped 1 previous similar message [ 1983.978476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.978963] Lustre: Skipped 1 previous similar message [ 1988.588102] LustreError: 124507:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 1988.624648] Lustre: server umount lustre-OST0000 complete [ 1991.648130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1991.650930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1991.654576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1991.656249] Lustre: Skipped 1 previous similar message [ 1994.208215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.208728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.967524] LustreError: 124714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1995.969223] LustreError: 124714:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1995.997100] Lustre: server umount lustre-MDT0000 complete [ 1997.215507] LustreError: 121563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878043 with bad export cookie 16001910866873262694 [ 1997.218828] LustreError: 121563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1997.219405] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.288338] LustreError: 124916:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1997.291941] LustreError: 124916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1997.347532] Lustre: server umount lustre-MDT0001 complete [ 2000.217249] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:07:25 (1763878045) [ 2002.440231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.633821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.943284] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2006.840910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2006.950788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2008.154936] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2009.886052] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2011.124423] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2011.774242] LustreError: 125617:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2011.777067] LustreError: 125617:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2011.779231] LustreError: 125617:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2012.129820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2012.129933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.130685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.130689] Lustre: Skipped 1 previous similar message [ 2012.140990] Lustre: Skipped 1 previous similar message [ 2017.916800] LustreError: 126928:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2017.918967] LustreError: 126928:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2017.944919] Lustre: server umount lustre-MDT0000 complete [ 2019.140752] LustreError: 125566:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878065 with bad export cookie 16001910866873263534 [ 2019.143403] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.143944] LustreError: 125566:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2019.257091] Lustre: server umount lustre-MDT0001 complete [ 2021.777879] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 01:07:47 (1763878067) [ 2023.929509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2039.391698] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2044.511441] LustreError: 127773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2070.111322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.973797] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2073.780611] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2074.506297] LustreError: 128437:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2074.508306] LustreError: 128437:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2074.547494] Lustre: server umount lustre-OST0000 complete [ 2077.279907] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 01:08:42 (1763878122) [ 2079.504781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.674996] LustreError: 129107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.680189] LustreError: 129107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2080.999161] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2083.801201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2085.161479] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2086.904466] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2088.062581] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2090.305325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2090.429143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.431155] Lustre: Skipped 2 previous similar messages [ 2092.329547] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2094.013815] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2114.527990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2114.530037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.533474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.534939] Lustre: Skipped 3 previous similar messages [ 2117.124277] LustreError: 131375:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2117.126292] LustreError: 131375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2117.154138] Lustre: server umount lustre-MDT0000 complete [ 2118.486090] LustreError: 129088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878164 with bad export cookie 16001910866873264178 [ 2118.491813] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.493634] LustreError: 129088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2118.619053] Lustre: server umount lustre-MDT0001 complete [ 2130.276412] Lustre: server umount lustre-OST0000 complete [ 2133.736358] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 01:09:39 (1763878179) [ 2135.915032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.082762] LustreError: 132449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.105541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.441342] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2140.344614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2141.797056] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2143.587023] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2144.749546] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2146.980426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2148.979129] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2150.746337] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2152.130749] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2161.449118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2162.457165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2163.656366] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2163.720953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2165.727806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2165.730056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.733347] Lustre: Skipped 2 previous similar messages [ 2165.734796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2165.736758] Lustre: Skipped 3 previous similar messages [ 2170.413242] LustreError: 135262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2170.415809] LustreError: 135262:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2170.449370] Lustre: server umount lustre-OST0000 complete [ 2176.480776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.484137] LustreError: Skipped 1 previous similar message [ 2178.204419] Lustre: server umount lustre-MDT0000 complete [ 2179.445215] LustreError: 133167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878225 with bad export cookie 16001910866873265235 [ 2179.448323] LustreError: 133167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2179.449114] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.561197] Lustre: server umount lustre-MDT0001 complete [ 2182.236817] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 01:10:27 (1763878227) [ 2184.349736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2199.775782] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2204.895457] LustreError: 136307:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2230.495352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.499126] Lustre: Skipped 2 previous similar messages [ 2232.476841] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2234.268190] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2236.407775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.552400] LustreError: 136334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.604026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2237.906503] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2240.633331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2241.993028] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2243.629794] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2244.893487] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2246.207031] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2246.280921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2247.628936] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2247.715746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2251.233788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.234716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2251.235486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.235492] Lustre: Skipped 4 previous similar messages [ 2251.239140] Lustre: Skipped 4 previous similar messages [ 2254.458386] LustreError: 139081:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2254.460732] LustreError: 139081:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2254.484786] Lustre: server umount lustre-MDT0000 complete [ 2255.648928] LustreError: 136314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878301 with bad export cookie 16001910866873266243 [ 2255.652296] LustreError: 136314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2255.652375] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.758101] Lustre: server umount lustre-MDT0001 complete [ 2266.142717] Lustre: server umount lustre-OST0000 complete [ 2268.862992] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 01:11:54 (1763878314) [ 2271.061908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2286.495440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2291.615331] LustreError: 140129:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2318.946027] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2320.545731] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2322.647607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.783048] LustreError: 140156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.832389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2324.010243] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2326.780694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2328.223710] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2329.885316] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2330.958972] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2333.125184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2333.156286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2333.209440] Lustre: lustre-OST0001: new disk, initializing [ 2333.211189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2334.959304] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2336.759401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2336.762318] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2336.789848] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2337.912094] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2339.096533] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2339.155959] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2340.331699] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2340.391596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2341.861957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.865289] Lustre: Skipped 2 previous similar messages [ 2341.866087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.867848] Lustre: Skipped 3 previous similar messages [ 2347.053129] LustreError: 143859:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2347.054709] LustreError: 143859:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2347.069325] Lustre: server umount lustre-OST0000 complete [ 2352.098774] LustreError: 142473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.098885] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2352.104562] LustreError: 142473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2352.106285] LustreError: Skipped 1 previous similar message [ 2354.389021] Lustre: server umount lustre-OST0001 complete [ 2361.752073] Lustre: server umount lustre-MDT0000 complete [ 2362.335943] LustreError: 141057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.340104] LustreError: 141057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2362.869188] LustreError: 140137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878408 with bad export cookie 16001910866873267188 [ 2362.871895] LustreError: 140137:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2362.871988] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.981070] Lustre: server umount lustre-MDT0001 complete [ 2364.658076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2365.889182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.138230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2368.417162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2370.377690] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 01:13:35 (1763878415) [ 2370.885231] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2371.430333] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2371.964862] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 01:13:37 (1763878417) [ 2374.136074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.241603] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2374.251704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2374.338643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.340473] Lustre: Skipped 6 previous similar messages [ 2375.733542] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2378.527786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2378.561581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2378.572795] Lustre: 146098:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2380.010446] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2381.744962] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2382.925546] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2385.208286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2385.291192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2387.278282] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2388.986540] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2390.316356] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2392.076095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2392.433494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2393.676858] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2393.747178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2400.525373] Lustre: server umount lustre-OST0000 complete [ 2414.127575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2416.076207] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2417.316735] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.320284] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2417.320337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.847394] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2419.309102] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2419.378510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2420.546109] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2420.604612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2421.660401] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e8410138000.ost_server_uuid 50 [ 2422.122470] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e8410138000.ost_server_uuid in FULL state after 0 sec [ 2422.174446] LustreError: 146105:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.203.8@tcp) returned error from blocking AST (req@00000000105a2ff0 x1849560128471936 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000076da4aa2/0xde123526d371bd39 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.8@tcp remote: 0x31b90640cc19cfdc expref: 5 pid: 146105 timeout: 2522 lvb_type: 0 [ 2422.188320] LustreError: lustre-MDT0000: A client on nid 192.168.203.8@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2422.192385] LustreError: 146093:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.8@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000076da4aa2/0xde123526d371bd39 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.8@tcp remote: 0x31b90640cc19cfdc expref: 6 pid: 146105 timeout: 0 lvb_type: 0 [ 2427.876258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.876459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.879728] Lustre: Skipped 7 previous similar messages [ 2427.881656] LustreError: Skipped 3 previous similar messages [ 2429.266211] Lustre: server umount lustre-OST0000 complete [ 2436.627715] Lustre: server umount lustre-MDT0000 complete [ 2437.826533] LustreError: 146826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878483 with bad export cookie 16001910866873268364 [ 2437.830620] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.015671] Lustre: server umount lustre-MDT0001 complete [ 2447.362143] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2448.306351] Key type lgssc unregistered [ 2448.425345] LNet: 151593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.427273] LNetError: 151593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2448.435292] LNet: Removed LNI 192.168.203.108@tcp [ 2448.711783] Key type .llcrypt unregistered [ 2448.712721] Key type ._llcrypt unregistered [ 2457.032099] Key type ._llcrypt registered [ 2457.033175] Key type .llcrypt registered [ 2457.070127] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2457.439827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.450176] alg: No test for adler32 (adler32-zlib) [ 2458.300921] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2458.376838] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2459.951139] Key type lgssc registered [ 2460.300185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2462.771746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2462.777490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.923853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.129644] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2467.802543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2467.909293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2469.118956] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2470.778928] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2471.918825] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2474.037494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2474.145797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.922370] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2477.661226] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2488.490174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2495.633979] Lustre: Failing over lustre-MDT0000 [ 2495.841092] LustreError: 155230:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2495.865722] Lustre: server umount lustre-MDT0000 complete [ 2498.527927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2498.530298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.536285] LustreError: 152963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.539731] LustreError: 152963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2499.040545] LustreError: 152963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.041655] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.047211] LustreError: 152963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2499.050010] Lustre: Skipped 1 previous similar message [ 2504.160145] LustreError: 152963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.166260] LustreError: 152963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2506.697431] LustreError: 152944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.8@tcp arrived at 1763878552 with bad export cookie 7188625837160580754 [ 2506.700728] LustreError: 152944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2508.922575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.955851] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.040606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.068172] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2510.258864] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2510.946275] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2510.948444] Lustre: lustre-MDT0000: Denying connection for new client d3e2741c-3e16-4902-85e8-93bfad935c09 (at 192.168.203.8@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2514.406228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2518.495994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2518.499417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2518.505351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.521633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2519.523168] Lustre: Skipped 1 previous similar message [ 2523.692092] LustreError: 156163:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2523.695388] LustreError: 156163:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2523.730061] Lustre: server umount lustre-OST0000 complete [ 2524.918822] LustreError: 156364:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2524.923854] Lustre: 155731:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2524.927704] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2524.935189] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2524.940898] LustreError: 155731:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008951153a x1849560666247552/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2524.949590] LustreError: 155731:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2524.954441] LustreError: 155731:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2524.959303] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2525.039827] LustreError: 156364:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2525.041575] LustreError: 156364:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2525.063238] Lustre: server umount lustre-MDT0000 complete [ 2526.216297] LustreError: 152944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878572 with bad export cookie 7188625837160581104 [ 2526.219983] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.220135] LustreError: 152272:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2532.351232] LustreError: 156565:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2532.353792] LustreError: 156565:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2532.409439] Lustre: server umount lustre-MDT0001 complete [ 2535.737181] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2536.733845] Key type lgssc unregistered [ 2536.865361] LNet: 157096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2536.867490] LNetError: 157096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2536.874273] LNet: Removed LNI 192.168.203.108@tcp [ 2537.173224] Key type .llcrypt unregistered [ 2537.174137] Key type ._llcrypt unregistered [ 2545.469129] Key type ._llcrypt registered [ 2545.470794] Key type .llcrypt registered [ 2545.508850] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2545.837751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.899727] alg: No test for adler32 (adler32-zlib) [ 2546.745024] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2546.819682] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2548.399136] Key type lgssc registered [ 2548.721665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.130086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.135803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.286362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.515079] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2556.201837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2556.317396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2557.613215] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2559.352769] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2560.477486] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2562.626383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2562.731763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.549899] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2566.253585] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2577.066694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2587.618140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.619368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2587.623868] Lustre: Skipped 1 previous similar message [ 2587.624236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2587.626589] Lustre: Skipped 1 previous similar message [ 2589.292074] LustreError: 160735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2589.324404] Lustre: server umount lustre-OST0000 complete [ 2592.223737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2592.226950] LustreError: Skipped 1 previous similar message [ 2592.228144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.231528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.665777] LustreError: 160939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2596.667352] LustreError: 160939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2596.691735] Lustre: server umount lustre-MDT0000 complete [ 2597.856297] LustreError: 159189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.860920] LustreError: 159189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2597.933588] LustreError: 158446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878643 with bad export cookie 15653641612539737468 [ 2597.937755] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.004163] LustreError: 161142:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2598.006909] LustreError: 161142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2598.063917] Lustre: server umount lustre-MDT0001 complete [ 2601.701696] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2602.687312] Key type lgssc unregistered [ 2602.812335] LNet: 161673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2602.814221] LNetError: 161673:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2602.825287] LNet: Removed LNI 192.168.203.108@tcp [ 2603.111111] Key type .llcrypt unregistered [ 2603.111980] Key type ._llcrypt unregistered [ 2614.848274] Key type ._llcrypt registered [ 2614.849206] Key type .llcrypt registered [ 2614.887366] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2615.258196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2615.344650] alg: No test for adler32 (adler32-zlib) [ 2616.195130] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2616.275675] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2617.855147] Key type lgssc registered [ 2618.237109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.817199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.824918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.986313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.211161] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2626.003333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2626.126203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.404580] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2629.141967] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2630.302519] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2632.437759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2632.552953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.305798] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2636.115677] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2647.907947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2657.248118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2657.251734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.258194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.273120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.273891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2658.275552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.078115] LustreError: 165598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2660.116444] Lustre: server umount lustre-OST0000 complete [ 2661.856059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2661.859992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.867099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.870233] Lustre: Skipped 1 previous similar message [ 2667.517190] LustreError: 165801:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2667.518711] LustreError: 165801:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2667.547950] Lustre: server umount lustre-MDT0000 complete [ 2668.512222] LustreError: 163332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.519454] LustreError: 163332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2668.755694] LustreError: 163311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878714 with bad export cookie 3186402874335398553 [ 2668.759251] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.826253] LustreError: 166005:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2668.828506] LustreError: 166005:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2668.883433] Lustre: server umount lustre-MDT0001 complete [ 2672.246464] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2673.270154] Key type lgssc unregistered [ 2673.430419] LNet: 166535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.433162] LNetError: 166535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2673.443330] LNet: Removed LNI 192.168.203.108@tcp [ 2673.749292] Key type .llcrypt unregistered [ 2673.750313] Key type ._llcrypt unregistered [ 2682.186672] Key type ._llcrypt registered [ 2682.187637] Key type .llcrypt registered [ 2682.228767] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2682.589595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2682.627052] alg: No test for adler32 (adler32-zlib) [ 2683.482695] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2683.578959] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2685.159167] Key type lgssc registered [ 2685.538987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2689.144859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2689.149811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.284923] Lustre: *** cfs_fail_loc=135, val=0*** [ 2690.333351] LustreError: 168016:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2690.336462] LustreError: 168016:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.203.108@tcp: cfg command failed: rc = -2 [ 2690.340545] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2690.340545] [ 2690.344716] LustreError: MGC192.168.203.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2690.352070] LustreError: 167976:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2690.356572] LustreError: 167976:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2690.360513] LustreError: 167976:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2690.364453] LustreError: 167976:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2690.395749] Lustre: server umount lustre-MDT0000 complete [ 2690.396878] LustreError: 167976:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2692.534223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.678335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.987924] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2696.846753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2696.974505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2698.268278] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2699.987961] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2701.156112] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2701.814351] LustreError: 168419:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2701.816919] LustreError: 168419:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2701.819013] LustreError: 168419:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2702.306243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.306369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2702.309989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.312349] Lustre: Skipped 1 previous similar message [ 2706.400967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2706.402738] Lustre: Skipped 1 previous similar message [ 2711.520643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.522342] Lustre: Skipped 1 previous similar message [ 2716.127153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2716.167867] LustreError: 169731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2716.171025] LustreError: 169731:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2716.206224] Lustre: server umount lustre-MDT0000 complete [ 2716.640266] LustreError: 168390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.644716] LustreError: 168390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2717.382331] LustreError: 168414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878763 with bad export cookie 10684308150199033344 [ 2717.387090] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.448858] LustreError: 169935:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2717.451706] LustreError: 169935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2717.509155] Lustre: server umount lustre-MDT0001 complete [ 2720.842786] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2721.886306] Key type lgssc unregistered [ 2722.014372] LNet: 170466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.016565] LNetError: 170466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.024414] LNet: Removed LNI 192.168.203.108@tcp [ 2722.322467] Key type .llcrypt unregistered [ 2722.323383] Key type ._llcrypt unregistered [ 2728.682387] Key type ._llcrypt registered [ 2728.683327] Key type .llcrypt registered [ 2728.723087] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2728.814261] Key type .llcrypt unregistered [ 2728.815175] Key type ._llcrypt unregistered [ 2735.189564] Key type ._llcrypt registered [ 2735.190472] Key type .llcrypt registered [ 2735.226121] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2735.526922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2735.584062] alg: No test for adler32 (adler32-zlib) [ 2736.442870] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2736.530470] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2738.111153] Key type lgssc registered [ 2738.451595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2740.962333] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2740.969652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2756.447303] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2761.567223] LustreError: 172533:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.108@tcp: failed processing log, type 4: rc = -110 [ 2788.191259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.115762] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2792.146147] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2794.752694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.922513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.952312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2796.203151] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2799.041936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2799.188639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.576965] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2802.326873] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2803.534965] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2804.496055] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2813.922530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2813.930573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.933246] Lustre: Skipped 1 previous similar message [ 2819.040907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.042656] Lustre: Skipped 1 previous similar message [ 2819.117048] LustreError: 175304:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2819.150425] Lustre: server umount lustre-OST0000 complete [ 2824.160700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.161706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.164690] Lustre: Skipped 2 previous similar messages [ 2826.433259] LustreError: 175505:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2826.436026] LustreError: 175505:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2826.460505] Lustre: server umount lustre-MDT0000 complete [ 2827.647763] LustreError: 174859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878873 with bad export cookie 12539272899310738320 [ 2827.650892] LustreError: 174859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2827.652559] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2827.723507] LustreError: 175707:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2827.726125] LustreError: 175707:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2827.793634] Lustre: server umount lustre-MDT0001 complete [ 2831.471441] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2832.593463] Key type lgssc unregistered [ 2832.727367] LNet: 176238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2832.729451] LNetError: 176238:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2832.738374] LNet: Removed LNI 192.168.203.108@tcp [ 2833.023564] Key type .llcrypt unregistered [ 2833.024516] Key type ._llcrypt unregistered [ 2841.829550] Key type ._llcrypt registered [ 2841.830473] Key type .llcrypt registered [ 2841.868424] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2842.221756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2842.247539] alg: No test for adler32 (adler32-zlib) [ 2843.109799] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2843.201843] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2844.791145] Key type lgssc registered [ 2845.134145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2847.677170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2847.685499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.842535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.229331] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2853.040487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2853.176202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.556244] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2856.295670] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2857.535829] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2859.786731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2859.894930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.699101] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2863.412706] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2873.191052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2875.012067] Lustre: Failing over lustre-MDT0000 [ 2875.083239] LustreError: 179925:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2875.108655] Lustre: server umount lustre-MDT0000 complete [ 2878.432922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.434292] LustreError: 177605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.435927] Lustre: Skipped 2 previous similar messages [ 2878.440480] LustreError: 177605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2879.048924] LustreError: 178322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.552075] LustreError: 178321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.558138] LustreError: 178321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2887.855667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.899867] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2888.019807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.048628] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2889.284387] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2889.383891] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2890.894700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2893.285056] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2893.290123] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2893.306539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2896.670403] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2905.055910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.057964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.061412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.642567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2908.643508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.645084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.645678] LustreError: Skipped 1 previous similar message [ 2908.656124] Lustre: Skipped 1 previous similar message [ 2909.935177] LustreError: 181254:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2909.937051] LustreError: 181254:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2909.972231] Lustre: server umount lustre-OST0000 complete [ 2913.761224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.762481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.763307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.770770] Lustre: Skipped 1 previous similar message [ 2917.308031] LustreError: 181460:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2917.309624] LustreError: 181460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2917.335262] Lustre: server umount lustre-MDT0000 complete [ 2918.509209] LustreError: 179928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763878964 with bad export cookie 11593918460421936774 [ 2918.512037] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2918.514827] LustreError: 179928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2918.623631] Lustre: server umount lustre-MDT0001 complete [ 2922.162098] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 2923.152947] Key type lgssc unregistered [ 2923.291350] LNet: 182191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2923.294518] LNetError: 182191:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2923.306417] LNet: Removed LNI 192.168.203.108@tcp [ 2923.623817] Key type .llcrypt unregistered [ 2923.625535] Key type ._llcrypt unregistered [ 2932.437539] Key type ._llcrypt registered [ 2932.438431] Key type .llcrypt registered [ 2932.469321] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 2932.781176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2932.799973] alg: No test for adler32 (adler32-zlib) [ 2933.646528] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 2933.721884] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 2935.303152] Key type lgssc registered [ 2935.646806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2938.110373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2938.116343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.269250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.519878] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2943.211627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.329217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.564052] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2946.282692] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2947.531986] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2949.746530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2949.868959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.705749] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 2953.415242] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2964.198351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 2969.720683] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 2980.473758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 2988.430911] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3000.289201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3000.289915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.291955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3000.292579] LustreError: Skipped 1 previous similar message [ 3000.299684] Lustre: Skipped 1 previous similar message [ 3003.693592] LustreError: 186022:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3003.729556] Lustre: server umount lustre-OST0000 complete [ 3005.409690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.410398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.415525] Lustre: Skipped 1 previous similar message [ 3005.416976] Lustre: Skipped 1 previous similar message [ 3010.528110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.529627] Lustre: Skipped 2 previous similar messages [ 3015.647916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.650434] Lustre: Skipped 1 previous similar message [ 3019.231119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.266811] LustreError: 186224:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3019.268357] LustreError: 186224:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3019.294347] Lustre: server umount lustre-MDT0000 complete [ 3020.442474] LustreError: 183541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879066 with bad export cookie 1013949316931772483 [ 3020.446982] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.522373] LustreError: 186429:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3020.523901] LustreError: 186429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3020.580977] Lustre: server umount lustre-MDT0001 complete [ 3023.904892] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3024.972633] Key type lgssc unregistered [ 3025.103435] LNet: 186960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.105413] LNetError: 186960:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.113411] LNet: Removed LNI 192.168.203.108@tcp [ 3025.424175] Key type .llcrypt unregistered [ 3025.425256] Key type ._llcrypt unregistered [ 3033.928607] Key type ._llcrypt registered [ 3033.929507] Key type .llcrypt registered [ 3033.963124] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3034.259047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3034.314780] alg: No test for adler32 (adler32-zlib) [ 3035.151660] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3035.226207] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3036.799142] Key type lgssc registered [ 3037.128538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3039.567078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3039.572166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.712401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.889105] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3044.540049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3044.654075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3045.888370] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3047.530934] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3048.663310] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3050.771813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3050.883946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.945907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3052.730429] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3054.475146] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3056.793152] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3064.375166] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3074.783098] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3083.606502] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3092.959775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.961538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.964504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.008469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.008479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.010571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.128840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.130551] Lustre: Skipped 2 previous similar messages [ 3104.223139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3104.239438] LustreError: 192545:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3104.279906] Lustre: server umount lustre-OST0000 complete [ 3106.271970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3106.273772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.279285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.368259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.368776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.548709] LustreError: 192748:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3111.552162] LustreError: 192748:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3111.583148] Lustre: server umount lustre-MDT0000 complete [ 3112.771181] LustreError: 188310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879158 with bad export cookie 5780016934253694982 [ 3112.773858] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3112.774316] LustreError: 188310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3112.847269] LustreError: 192950:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3112.848795] LustreError: 192950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3112.897754] Lustre: server umount lustre-MDT0001 complete [ 3116.362957] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3117.417190] Key type lgssc unregistered [ 3117.555426] LNet: 193480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3117.557552] LNetError: 193480:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3117.568374] LNet: Removed LNI 192.168.203.108@tcp [ 3117.903470] Key type .llcrypt unregistered [ 3117.904509] Key type ._llcrypt unregistered [ 3126.391080] Key type ._llcrypt registered [ 3126.392179] Key type .llcrypt registered [ 3126.433547] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3126.737233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3126.746991] alg: No test for adler32 (adler32-zlib) [ 3127.615054] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3127.698973] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3129.279179] Key type lgssc registered [ 3129.728287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3132.868718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3132.878589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.039951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.208173] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3138.354184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3138.478443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.812483] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3141.662892] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3142.828223] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3145.060774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3145.199379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.343853] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3149.197116] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3159.528451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3163.615910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3163.619708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.625024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3164.641107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3164.641807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3164.641854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.642894] Lustre: Skipped 1 previous similar message [ 3165.292987] LustreError: 197266:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3165.324143] Lustre: server umount lustre-OST0000 complete [ 3168.894793] LustreError: 197467:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3168.897676] LustreError: 197467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3168.924815] Lustre: server umount lustre-MDT0000 complete [ 3169.248532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3169.251237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3169.255670] LustreError: 194853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.260744] LustreError: 194853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3169.760815] LustreError: 194849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.224521] LustreError: 194833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879216 with bad export cookie 5784669205629984574 [ 3170.228192] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3170.305234] LustreError: 197669:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3170.307033] LustreError: 197669:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3170.353115] Lustre: server umount lustre-MDT0001 complete [ 3174.498342] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3175.857568] Key type lgssc unregistered [ 3175.984358] LNet: 198200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3175.986519] LNetError: 198200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3175.993316] LNet: Removed LNI 192.168.203.108@tcp [ 3176.282261] Key type .llcrypt unregistered [ 3176.283786] Key type ._llcrypt unregistered [ 3183.990079] Key type ._llcrypt registered [ 3183.990914] Key type .llcrypt registered [ 3184.023836] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3184.360076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3184.399526] alg: No test for adler32 (adler32-zlib) [ 3185.256441] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3185.336583] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3186.919146] Key type lgssc registered [ 3187.247734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3189.803255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3189.808884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.959205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3192.167677] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3194.901041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3195.028901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3196.446079] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3198.283303] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3199.402768] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3201.622132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3201.738156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.567370] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3205.342828] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3215.080668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3220.450416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3220.453544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.455430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3220.460344] Lustre: Skipped 1 previous similar message [ 3220.467728] Lustre: Skipped 1 previous similar message [ 3221.293313] LustreError: 201843:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3221.326418] Lustre: server umount lustre-OST0000 complete [ 3224.699190] LustreError: 202045:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3224.700733] LustreError: 202045:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3224.723588] Lustre: server umount lustre-MDT0000 complete [ 3225.569138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.570184] LustreError: 199423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.574722] Lustre: Skipped 1 previous similar message [ 3225.580067] LustreError: 199423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3225.949671] LustreError: 199406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879271 with bad export cookie 8782737193785229449 [ 3225.954363] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3226.028927] LustreError: 202247:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3226.030911] LustreError: 202247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3226.094168] Lustre: server umount lustre-MDT0001 complete [ 3229.924667] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3231.076450] Key type lgssc unregistered [ 3231.217670] LNet: 202777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.220763] LNetError: 202777:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.229614] LNet: Removed LNI 192.168.203.108@tcp [ 3231.619166] Key type .llcrypt unregistered [ 3231.621233] Key type ._llcrypt unregistered [ 3238.806929] Key type ._llcrypt registered [ 3238.809502] Key type .llcrypt registered [ 3238.846968] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3239.151512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3239.197851] alg: No test for adler32 (adler32-zlib) [ 3240.035407] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3240.110628] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3241.687215] Key type lgssc registered [ 3242.034719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3244.872686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3244.882818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.052317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.387027] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3250.376457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3250.543682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3251.997393] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3254.067768] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3255.194190] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3258.194768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3258.330948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3260.655365] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3263.109496] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3271.653545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3276.564485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3276.587043] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3276.657057] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3279.031344] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3281.383703] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3282.083153] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3292.811738] Lustre: Permanently deactivating lustre-OST0001 [ 3292.820690] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3295.808707] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3302.839734] Lustre: Permanently reactivating lustre-OST0001 [ 3302.849555] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3302.853238] Lustre: Skipped 2 previous similar messages [ 3309.062225] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.068934] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3309.072604] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3309.080594] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3309.099653] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3314.144168] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.144238] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.145886] LustreError: Skipped 1 previous similar message [ 3314.146333] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3314.152815] Lustre: Skipped 2 previous similar messages [ 3316.708079] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3316.709567] Lustre: Skipped 2 previous similar messages [ 3318.383240] LustreError: 207472:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3318.403557] Lustre: server umount lustre-OST0001 complete [ 3321.829946] LustreError: 205600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.831181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.831260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3321.832390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.839641] LustreError: 205600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3321.846457] Lustre: Skipped 1 previous similar message [ 3325.808448] LustreError: 207678:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3325.813212] LustreError: 207678:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3325.861134] Lustre: server umount lustre-OST0000 complete [ 3327.455867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3327.457878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.461644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.464613] Lustre: Skipped 1 previous similar message [ 3332.065127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3332.066136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.266639] LustreError: 207880:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3333.271656] LustreError: 207880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3333.312455] Lustre: server umount lustre-MDT0000 complete [ 3335.015725] LustreError: 207202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879381 with bad export cookie 9568794503119838092 [ 3335.021200] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3335.022393] LustreError: 207202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3335.178435] Lustre: server umount lustre-MDT0001 complete [ 3339.548029] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3340.818771] Key type lgssc unregistered [ 3340.969647] LNet: 208613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3340.972650] LNetError: 208613:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3340.986562] LNet: Removed LNI 192.168.203.108@tcp [ 3341.377925] Key type .llcrypt unregistered [ 3341.379540] Key type ._llcrypt unregistered [ 3352.784031] Key type ._llcrypt registered [ 3352.784913] Key type .llcrypt registered [ 3352.819931] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3358.283022] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3358.685434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3358.697333] alg: No test for adler32 (adler32-zlib) [ 3359.534900] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3359.614241] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3361.191140] Key type lgssc registered [ 3361.542093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3364.078935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3366.001054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3367.845870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3369.700271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3372.075523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3372.100402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3372.105618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.196940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3373.209596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3373.250936] Lustre: lustre-MDT0000: new disk, initializing [ 3373.276477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3373.281886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3374.848731] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3380.076038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3380.100169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3380.134365] Lustre: 212530:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3380.157303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3380.159286] Lustre: Skipped 1 previous similar message [ 3380.196483] Lustre: lustre-MDT0001: new disk, initializing [ 3380.232254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.244457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3380.248618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3381.461834] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3384.608252] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3385.996656] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3388.632692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3388.665100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3388.806523] Lustre: lustre-OST0000: new disk, initializing [ 3388.809038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3388.856897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.246473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3391.250851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3391.260138] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3391.295676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3394.808841] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.246252] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3396.319956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3397.764236] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3397.839185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3401.697763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.700880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.702262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.707104] Lustre: Skipped 1 previous similar message [ 3404.530183] LustreError: 215465:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3404.577220] Lustre: server umount lustre-OST0000 complete [ 3406.816891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.817957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3406.820952] Lustre: Skipped 1 previous similar message [ 3406.822600] Lustre: Skipped 1 previous similar message [ 3411.936920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.940139] Lustre: Skipped 2 previous similar messages [ 3412.350463] LustreError: 215668:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3412.353458] LustreError: 215668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3412.384505] Lustre: server umount lustre-MDT0000 complete [ 3413.700879] LustreError: 213309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879459 with bad export cookie 10849755180071062495 [ 3413.706552] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.707438] LustreError: 213309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3413.788605] LustreError: 215870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3413.790697] LustreError: 215870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3413.844321] Lustre: server umount lustre-MDT0001 complete [ 3417.235478] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 01:31:02 (1763879462) [ 3420.174278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.428351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.971355] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3425.613912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3425.725406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.316838] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3429.570239] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3431.136222] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3433.892025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3434.015745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.198219] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3438.380514] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3449.491454] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3454.988876] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3462.708772] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3468.440070] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3481.748272] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3481.752497] Lustre: Skipped 1 previous similar message [ 3499.152844] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3499.155813] Lustre: Skipped 1 previous similar message [ 3539.943536] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3539.948214] Lustre: Skipped 4 previous similar messages [ 3611.915814] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3611.917938] Lustre: Skipped 8 previous similar messages [ 3614.178739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.179441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3614.180064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.182486] Lustre: Skipped 1 previous similar message [ 3614.186576] LustreError: Skipped 2 previous similar messages [ 3619.296498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3619.297983] Lustre: Skipped 2 previous similar messages [ 3619.698194] LustreError: 220034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3619.701929] LustreError: 220034:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3619.757262] Lustre: server umount lustre-OST0000 complete [ 3624.416660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.419755] Lustre: Skipped 1 previous similar message [ 3627.083445] LustreError: 220240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3627.087014] LustreError: 220240:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3627.121444] Lustre: server umount lustre-MDT0000 complete [ 3628.718111] LustreError: 218159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879674 with bad export cookie 10849755180071063370 [ 3628.725933] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3628.732244] LustreError: 218159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3628.880330] Lustre: server umount lustre-MDT0001 complete [ 3632.860279] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3634.054426] Key type lgssc unregistered [ 3634.176440] LNet: 220973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3634.178661] LNetError: 220973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3634.188414] LNet: Removed LNI 192.168.203.108@tcp [ 3634.501270] Key type .llcrypt unregistered [ 3634.502400] Key type ._llcrypt unregistered [ 3644.315314] Key type ._llcrypt registered [ 3644.316600] Key type .llcrypt registered [ 3644.360178] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3644.686163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3644.693538] alg: No test for adler32 (adler32-zlib) [ 3645.533582] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3645.606298] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3647.175171] Key type lgssc registered [ 3647.627930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3650.880074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3650.886381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.033598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.569181] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3656.537896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3656.649979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3658.135052] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3660.386581] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3662.028127] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3664.644522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3664.784135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.969757] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_default_debug -1 all [ 3668.862266] Lustre: DEBUG MARKER: oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3687.394993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.395644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3687.398662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3687.400452] LustreError: Skipped 1 previous similar message [ 3687.402991] Lustre: Skipped 1 previous similar message [ 3691.186096] LustreError: 224775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3691.222844] Lustre: server umount lustre-OST0000 complete [ 3692.513162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.513270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3692.513971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.518370] Lustre: Skipped 2 previous similar messages [ 3697.631964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3697.633438] Lustre: Skipped 2 previous similar messages [ 3698.561343] LustreError: 224980:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3698.563364] LustreError: 224980:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3698.599703] Lustre: server umount lustre-MDT0000 complete [ 3699.829794] LustreError: 223977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763879745 with bad export cookie 15583171639227765940 [ 3699.831958] LustreError: MGC192.168.203.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.833212] LustreError: 223977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3699.906065] LustreError: 225182:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3699.907683] LustreError: 225182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3699.962862] Lustre: server umount lustre-MDT0001 complete [ 3703.142057] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3704.122782] Key type lgssc unregistered [ 3704.248321] LNet: 225712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3704.250480] LNetError: 225712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3704.259281] LNet: Removed LNI 192.168.203.108@tcp [ 3704.506882] Key type .llcrypt unregistered [ 3704.507753] Key type ._llcrypt unregistered [ 3711.614967] Key type ._llcrypt registered [ 3711.616204] Key type .llcrypt registered [ 3711.651690] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3711.753212] Key type .llcrypt unregistered [ 3711.754816] Key type ._llcrypt unregistered [ 3721.242911] Key type ._llcrypt registered [ 3721.244381] Key type .llcrypt registered [ 3721.296393] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing set_hostid [ 3727.961609] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing load_modules_local [ 3728.242188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3728.250743] alg: No test for adler32 (adler32-zlib) [ 3729.112279] Lustre: Lustre: Build Version: 2.16.61_40_gf2aba0f [ 3729.196128] LNet: Added LNI 192.168.203.108@tcp [8/256/0/180] [ 3730.775213] Key type lgssc registered [ 3731.238228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3734.228359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3737.005457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3739.598590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3742.108778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3742.662303] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3618 sec ========== 01:36:28 (1763879788) [ 3743.174389] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:36:28 (1763879788) === [ 3744.199577] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:36:29 (1763879789) === [ 3756.985692] Lustre: DEBUG MARKER: oleg308-server.virtnet: executing unload_modules_local [ 3758.194175] Key type lgssc unregistered [ 3758.360564] LNet: 231788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3758.367143] LNetError: 231788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3758.380536] LNet: Removed LNI 192.168.203.108@tcp [ 3758.860588] Key type .llcrypt unregistered [ 3758.862349] Key type ._llcrypt unregistered