[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 535471689 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002407] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007302] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011067] LSM: Security Framework initializing [ 0.012046] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014054] *** VALIDATE selinux *** [ 0.022237] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027657] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030089] *** VALIDATE tmpfs *** [ 0.032368] *** VALIDATE proc *** [ 0.033201] *** VALIDATE cgroup *** [ 0.034005] *** VALIDATE cgroup2 *** [ 0.036038] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037131] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039023] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.043574] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046822] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047017] ... version: 2 [ 0.048009] ... bit width: 48 [ 0.049008] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051009] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.053008] ... event mask: 000000070000000f [ 0.054305] rcu: Hierarchical SRCU implementation. [ 0.056440] smp: Bringing up secondary CPUs ... [ 0.057504] x86: Booting SMP configuration: [ 0.058020] .... node #0, CPUs: #1 #2 #3 [ 0.071079] smp: Brought up 1 node, 4 CPUs [ 0.073013] smpboot: Max logical packages: 1 [ 0.074013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.115179] node 0 deferred pages initialised in 39ms [ 0.119457] devtmpfs: initialized [ 0.120201] x86/mm: Memory block size: 128MB [ 0.123492] gcov: version magic: 0x41383552 [ 0.126277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134351] pinctrl core: initialized pinctrl subsystem [ 0.136144] [ 0.136905] ************************************************************* [ 0.140012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.143009] ** ** [ 0.145010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148010] ** ** [ 0.151016] ** This means that this kernel is built to expose internal ** [ 0.153009] ** IOMMU data structures, which may compromise security on ** [ 0.156016] ** your system. ** [ 0.159011] ** ** [ 0.161011] ** If you see this message and you are not debugging the ** [ 0.164011] ** kernel, report this immediately to your vendor! ** [ 0.167009] ** ** [ 0.170042] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ************************************************************* [ 0.175701] NET: Registered protocol family 16 [ 0.178609] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188693] cpuidle: using governor menu [ 0.190978] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193621] PCI: Using configuration type 1 for base access [ 0.196205] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209042] cryptd: max_cpu_qlen set to 1000 [ 0.210211] ACPI: Added _OSI(Module Device) [ 0.212013] ACPI: Added _OSI(Processor Device) [ 0.214011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.216014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230645] ACPI: Interpreter enabled [ 0.232057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234011] ACPI: Using IOAPIC for interrupt routing [ 0.236104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.240505] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253225] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259136] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269819] acpiphp: Slot [2] registered [ 0.272108] acpiphp: Slot [5] registered [ 0.273205] acpiphp: Slot [6] registered [ 0.275258] acpiphp: Slot [7] registered [ 0.278123] acpiphp: Slot [8] registered [ 0.279097] acpiphp: Slot [9] registered [ 0.281095] acpiphp: Slot [10] registered [ 0.283090] acpiphp: Slot [3] registered [ 0.284098] acpiphp: Slot [4] registered [ 0.286098] acpiphp: Slot [11] registered [ 0.288092] acpiphp: Slot [12] registered [ 0.290091] acpiphp: Slot [13] registered [ 0.291000] acpiphp: Slot [14] registered [ 0.292007] acpiphp: Slot [15] registered [ 0.294098] acpiphp: Slot [16] registered [ 0.296090] acpiphp: Slot [17] registered [ 0.297207] acpiphp: Slot [18] registered [ 0.299103] acpiphp: Slot [19] registered [ 0.300216] acpiphp: Slot [20] registered [ 0.302108] acpiphp: Slot [21] registered [ 0.304103] acpiphp: Slot [22] registered [ 0.306154] acpiphp: Slot [23] registered [ 0.308094] acpiphp: Slot [24] registered [ 0.310097] acpiphp: Slot [25] registered [ 0.312094] acpiphp: Slot [26] registered [ 0.313235] acpiphp: Slot [27] registered [ 0.315217] acpiphp: Slot [28] registered [ 0.316110] acpiphp: Slot [29] registered [ 0.318100] acpiphp: Slot [30] registered [ 0.320093] acpiphp: Slot [31] registered [ 0.322112] PCI host bridge to bus 0000:00 [ 0.324018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.340019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343486] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349571] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355664] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380046] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.390015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392724] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395925] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402924] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.430011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.437170] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.444014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.468016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476580] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.482017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.489014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.504016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510937] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.521017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549687] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.561015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.576014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.586491] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.595015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612016] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.616012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.620013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.646147] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.649470] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.652451] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.655494] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.658316] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.664239] iommu: Default domain type: Passthrough [ 0.665270] SCSI subsystem initialized [ 0.667147] ACPI: bus type USB registered [ 0.668000] usbcore: registered new interface driver usbfs [ 0.669072] usbcore: registered new interface driver hub [ 0.671075] usbcore: registered new device driver usb [ 0.673157] pps_core: LinuxPPS API ver. 1 registered [ 0.674007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676055] PTP clock support registered [ 0.678092] EDAC MC: Ver: 3.0.0 [ 0.679221] PCI: Using ACPI for IRQ routing [ 0.680557] NetLabel: Initializing [ 0.682013] NetLabel: domain hash size = 128 [ 0.684011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.686101] NetLabel: unlabeled traffic allowed by default [ 0.688331] vgaarb: loaded [ 0.691380] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.700303] clocksource: Switched to clocksource kvm-clock [ 0.823645] VFS: Disk quotas dquot_6.6.0 [ 0.824629] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.826086] *** VALIDATE ramfs *** [ 0.827188] *** VALIDATE hugetlbfs *** [ 0.828255] pnp: PnP ACPI init [ 0.829743] pnp: PnP ACPI: found 6 devices [ 0.845072] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.849622] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861746] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.867132] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.871593] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.877577] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.882380] NET: Registered protocol family 2 [ 0.886225] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.894766] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.898535] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.914321] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.921686] TCP: Hash tables configured (established 65536 bind 65536) [ 0.925801] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.929713] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.933389] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936858] NET: Registered protocol family 1 [ 0.940565] RPC: Registered named UNIX socket transport module. [ 0.943679] RPC: Registered udp transport module. [ 0.945764] RPC: Registered tcp transport module. [ 0.947696] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.950200] NET: Registered protocol family 44 [ 0.952985] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.955273] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.957834] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.961339] PCI: CLS 0 bytes, default 64 [ 0.964354] Unpacking initramfs... [ 2.795572] debug: unmapping init [mem 0xffff933f7cc54000-0xffff933f7ffbffff] [ 2.802439] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.806308] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.811346] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.488703] Initialise system trusted keyrings [ 3.493325] Key type blacklist registered [ 3.496107] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.507453] zbud: loaded [ 3.511369] *** VALIDATE nfs *** [ 3.512655] *** VALIDATE nfs4 *** [ 3.514630] pstore: using deflate compression [ 3.519233] Platform Keyring initialized [ 3.646120] NET: Registered protocol family 38 [ 3.648343] Key type asymmetric registered [ 3.650759] Asymmetric key parser 'x509' registered [ 3.653084] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.656773] io scheduler mq-deadline registered [ 3.658807] io scheduler kyber registered [ 3.668474] io scheduler bfq registered [ 3.672695] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.678759] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.682534] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.686420] ACPI: Power Button [PWRF] [ 3.693076] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.701393] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.716688] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.726235] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.755246] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.786355] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.822501] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.829660] Non-volatile memory driver v1.3 [ 3.831770] Linux agpgart interface v0.103 [ 3.960532] virtio_blk virtio1: [vda] 133608 512-byte logical blocks (68.4 MB/65.2 MiB) [ 3.964808] vda: detected capacity change from 0 to 68407296 [ 3.996428] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.033047] vdb: detected capacity change from 0 to 1073741824 [ 4.103489] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.115053] vdc: detected capacity change from 0 to 2621440000 [ 4.176027] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.183858] vdd: detected capacity change from 0 to 2621440000 [ 4.207684] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.211118] vde: detected capacity change from 0 to 4294967296 [ 4.227066] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.230164] vdf: detected capacity change from 0 to 4294967296 [ 4.238983] libphy: Fixed MDIO Bus: probed [ 4.263473] usbcore: registered new interface driver usbserial_generic [ 4.266538] usbserial: USB Serial support registered for generic [ 4.269227] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.275268] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.277815] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.281182] mousedev: PS/2 mouse device common for all mice [ 4.286698] rtc_cmos 00:05: RTC can wake from S4 [ 4.292676] rtc_cmos 00:05: registered as rtc0 [ 4.295637] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.300074] intel_pstate: CPU model not supported [ 4.302848] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.308138] hid: raw HID events driver (C) Jiri Kosina [ 4.313604] usbcore: registered new interface driver usbhid [ 4.314168] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.319530] usbhid: USB HID core driver [ 4.319739] drop_monitor: Initializing network drop monitor service [ 4.321018] Initializing XFRM netlink socket [ 4.330137] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.339731] NET: Registered protocol family 10 [ 4.343101] Segment Routing with IPv6 [ 4.345072] NET: Registered protocol family 17 [ 4.347535] mpls_gso: MPLS GSO support [ 4.547368] RAS: Correctable Errors collector initialized. [ 4.627447] AVX version of gcm_enc/dec engaged. [ 4.628756] AES CTR mode by8 optimization enabled [ 4.705428] sched_clock: Marking stable (4705007186, 0)->(5631204353, -926197167) [ 4.710593] registered taskstats version 1 [ 4.712509] Loading compiled-in X.509 certificates [ 4.714540] zswap: loaded using pool lzo/zbud [ 4.741540] Key type big_key registered [ 4.757288] Key type encrypted registered [ 4.758565] ima: No TPM chip found, activating TPM-bypass! [ 4.761334] ima: Allocated hash algorithm: sha1 [ 4.762529] ima: No architecture policies found [ 4.764220] evm: Initialising EVM extended attributes: [ 4.765851] evm: security.selinux [ 4.766801] evm: security.ima [ 4.767793] evm: security.capability [ 4.769139] evm: HMAC attrs: 0x1 [ 4.794297] rtc_cmos 00:05: setting system clock to 2026-03-11 13:14:18 UTC (1773234858) [ 4.805726] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 4.811129] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 4.821229] Write protecting the kernel read-only data: 28672k [ 4.827666] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 4.831097] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 4.904245] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.914442] systemd[1]: Detected virtualization kvm. [ 4.923588] systemd[1]: Detected architecture x86-64. [ 4.925766] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.952337] systemd[1]: No hostname configured. [ 4.957394] systemd[1]: Set hostname to . [ 4.959417] random: systemd: uninitialized urandom read (16 bytes read) [ 4.961670] systemd[1]: Initializing machine ID from random generator. [ 5.072335] random: ln: uninitialized urandom read (6 bytes read) [ 5.492701] random: systemd: uninitialized urandom read (16 bytes read) [ 5.496240] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.500839] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.505806] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.394665] device-mapper: uevent: version 1.0.3 [ 7.397689] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Mounting Kernel Configuration File System... Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 9.039503] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.449076] virtio_net virtio0 ens2: renamed from eth0 [ 9.704035] scsi host0: ata_piix [ 9.748599] scsi host1: ata_piix [ 9.751027] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.756382] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.433956] random: crng init done [ 14.435742] random: 7 urandom warning(s) missed due to ratelimiting [ 16.803591] dracut-initqueue[570]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.430319] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.860173] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.799863] SELinux: Disabled at runtime. [ 21.948814] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.978110] systemd[1]: Detected virtualization kvm. [ 21.991530] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.375720] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.380854] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.393124] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.430283] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.433125] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.439284] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.448244] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [[ 23.727857] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.653581] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.787482] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.881642] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.684018] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.809769] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.539470] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.516786] NFS: Registering the id_resolver key type [ 32.528248] Key type id_resolver registered [ 32.535754] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg234-server login: [ 71.065008] hrtimer: interrupt took 1281099 ns [ 75.317602] libcfs: loading out-of-tree module taints kernel. [ 75.330125] Key type ._llcrypt registered [ 75.331915] Key type .llcrypt registered [ 75.382542] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 85.978253] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 86.858214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.866197] alg: No test for adler32 (adler32-zlib) [ 87.969174] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 88.389916] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 90.061351] Key type lgssc registered [ 90.891440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.300685] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.282946] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.435622] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.953906] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.745862] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.463598] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 132.958201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.003405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 133.017408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.170945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 134.196318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 134.250258] Lustre: lustre-MDT0000: new disk, initializing [ 134.336937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.357366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.003541] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.543150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.583594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 145.629290] Lustre: 6472:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 145.664117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 145.668115] Lustre: Skipped 1 previous similar message [ 145.724994] Lustre: lustre-MDT0001: new disk, initializing [ 145.768949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 145.796331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 145.803068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 147.928994] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.899425] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 155.782200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.835635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 155.945645] Lustre: lustre-OST0000: new disk, initializing [ 155.951668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 155.993923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.740226] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.187352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 159.191817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 159.263439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 165.974237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.023342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 166.080844] Lustre: lustre-OST0001: new disk, initializing [ 166.083572] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 166.113760] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.190048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 167.196636] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 167.228694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 168.902614] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.026259] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 181.013773] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 189.148755] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing check_logdir /tmp/testlogs/ [ 192.070813] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing yml_node [ 194.639494] Lustre: DEBUG MARKER: Client: 2.16.59.50 [ 196.252727] Lustre: DEBUG MARKER: MDS: 2.16.59.50 [ 197.850767] Lustre: DEBUG MARKER: OSS: 2.16.59.50 [ 198.728662] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Mar 11 09:17:31 EDT 2026 [ 207.671984] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 208.475769] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 222.687830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 222.693844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.701662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 223.712431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.714161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 223.725350] Lustre: Skipped 1 previous similar message [ 223.738406] Lustre: Skipped 3 previous similar messages [ 228.218457] LustreError: 12174:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 228.352581] Lustre: server umount lustre-MDT0000 complete [ 230.749543] LustreError: 6464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235084 with bad export cookie 11979418718943917076 [ 230.754182] LustreError: 6464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 230.755445] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 233.951957] LustreError: 9997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 233.953107] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 233.954440] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 233.980094] LustreError: 9997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 233.990205] Lustre: Skipped 2 previous similar messages [ 236.000184] LustreError: 12067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 236.001277] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 236.012635] LustreError: 12067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 236.020266] Lustre: Skipped 2 previous similar messages [ 241.120268] LustreError: 6478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 241.120751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 241.126989] LustreError: 6478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 241.134472] Lustre: Skipped 1 previous similar message [ 245.216032] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 245.265509] LustreError: 12376:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 245.267802] LustreError: 12376:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 245.349669] Lustre: server umount lustre-MDT0001 complete [ 247.578101] LustreError: 12579:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 247.580808] LustreError: 12579:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 247.610099] Lustre: server umount lustre-OST0000 complete [ 249.777854] LustreError: 12779:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 249.781036] LustreError: 12779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 249.842743] Lustre: server umount lustre-OST0001 complete [ 253.335263] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 258.628408] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 264.437871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 268.451219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 272.438701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 276.446624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 280.456754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 280.493487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.657070] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 280.682763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 280.730513] Lustre: lustre-MDT0000: new disk, initializing [ 280.778049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 280.794804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 283.051714] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 289.170360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 289.206278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 289.275761] Lustre: 14967:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 289.315296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 289.319558] Lustre: Skipped 1 previous similar message [ 289.378897] Lustre: lustre-MDT0001: new disk, initializing [ 289.447131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 289.469133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 289.480720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 291.912285] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 296.548761] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 298.512321] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 302.042560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 302.088562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 303.393076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 303.460336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 305.559597] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 310.359496] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 312.714229] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 312.843651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 315.136058] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 315.246503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 318.944710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 318.950329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.953224] LustreError: Skipped 1 previous similar message [ 318.957556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 318.964043] Lustre: Skipped 1 previous similar message [ 318.974325] Lustre: Skipped 1 previous similar message [ 322.364283] LustreError: 17906:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 322.367841] LustreError: 17906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 322.439627] Lustre: server umount lustre-OST0000 complete [ 325.600078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 325.603763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.698413] LustreError: 18108:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 330.701155] LustreError: 18108:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 330.747781] Lustre: server umount lustre-MDT0000 complete [ 332.662580] LustreError: 14959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235186 with bad export cookie 11979418718943918609 [ 332.664239] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.672295] LustreError: 14959:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 332.871321] Lustre: server umount lustre-MDT0001 complete [ 334.766875] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:19:48 (1773235188) [ 338.160963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.403758] LustreError: 18836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.449921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.454781] Lustre: Skipped 1 previous similar message [ 340.797276] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 343.522339] LustreError: 18836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.259971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 347.619429] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 350.727909] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 352.848383] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 356.875849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 360.219860] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 363.800428] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 379.872599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 379.881591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.893208] Lustre: Skipped 1 previous similar message [ 379.898385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 379.903089] Lustre: Skipped 3 previous similar messages [ 391.135522] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 391.171114] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 391.174621] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 391.221963] Lustre: server umount lustre-OST0000 complete [ 396.257171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.264957] Lustre: Skipped 2 previous similar messages [ 399.395576] Lustre: server umount lustre-MDT0000 complete [ 401.375919] LustreError: 19870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.393522] LustreError: 19870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 401.404990] LustreError: 18815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235255 with bad export cookie 11979418718943919491 [ 401.409146] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 401.410777] LustreError: 18815:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 401.640339] Lustre: server umount lustre-MDT0001 complete [ 406.399682] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 407.901069] Key type lgssc unregistered [ 408.061050] LNet: 22040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.066777] LNetError: 22040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.074828] LNet: Removed LNI 192.168.202.134@tcp [ 408.493126] Key type .llcrypt unregistered [ 408.494666] Key type ._llcrypt unregistered [ 420.652898] Key type ._llcrypt registered [ 420.654215] Key type .llcrypt registered [ 420.713955] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 421.351676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 421.374691] alg: No test for adler32 (adler32-zlib) [ 422.308861] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 422.423723] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 424.023749] Key type lgssc registered [ 424.597203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 428.720317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 428.740087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.981264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.880481] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 436.840397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 437.036199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.953836] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 442.006517] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 444.041706] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 448.246110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 448.515629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.904634] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 454.698398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 455.329575] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.864414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 464.868622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.875560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.914277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.915100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 466.919244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.192142] LustreError: 26098:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 470.295097] Lustre: server umount lustre-OST0000 complete [ 473.058909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 473.064424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.072168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 473.076290] Lustre: Skipped 1 previous similar message [ 477.153944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.159119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.168757] Lustre: Skipped 1 previous similar message [ 478.602879] LustreError: 26300:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 478.605921] LustreError: 26300:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 478.641965] Lustre: server umount lustre-MDT0000 complete [ 480.710193] LustreError: 23392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235334 with bad export cookie 1622371115012352539 [ 480.729594] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.819700] LustreError: 26501:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 480.822299] LustreError: 26501:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 480.917589] Lustre: server umount lustre-MDT0001 complete [ 486.162703] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 487.743333] Key type lgssc unregistered [ 487.970899] LNet: 27032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.978718] LNetError: 27032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 487.997211] LNet: Removed LNI 192.168.202.134@tcp [ 488.539265] Key type .llcrypt unregistered [ 488.541051] Key type ._llcrypt unregistered [ 502.398182] Key type ._llcrypt registered [ 502.400991] Key type .llcrypt registered [ 502.469725] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 503.060741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 503.137290] alg: No test for adler32 (adler32-zlib) [ 504.086739] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 504.232671] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 505.831326] Key type lgssc registered [ 506.551294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 511.512083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 511.534288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.800655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.210807] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 519.970658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 520.151203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 522.472174] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 525.687394] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 528.194926] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 536.284256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 536.465855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.511367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 539.327491] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 541.925691] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 552.928755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.932809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.943673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 554.976974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 554.977799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 554.980875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.992267] Lustre: Skipped 1 previous similar message [ 557.497171] LustreError: 30995:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 557.576684] Lustre: server umount lustre-OST0000 complete [ 560.106240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.112410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 560.123356] Lustre: Skipped 1 previous similar message [ 560.130959] Lustre: Skipped 1 previous similar message [ 565.217829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 565.222158] Lustre: Skipped 1 previous similar message [ 565.656195] LustreError: 31196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 565.661289] LustreError: 31196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 565.756273] Lustre: server umount lustre-MDT0000 complete [ 567.784892] LustreError: 28384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235421 with bad export cookie 253224381902075450 [ 567.792940] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.800301] LustreError: 28384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 567.895943] LustreError: 31397:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 567.900343] LustreError: 31397:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 568.088245] Lustre: server umount lustre-MDT0001 complete [ 572.735891] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 574.167486] Key type lgssc unregistered [ 574.331959] LNet: 31929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.339876] LNetError: 31929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.350086] LNet: Removed LNI 192.168.202.134@tcp [ 574.806077] Key type .llcrypt unregistered [ 574.808324] Key type ._llcrypt unregistered [ 586.188391] Key type ._llcrypt registered [ 586.189722] Key type .llcrypt registered [ 586.231544] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 586.692363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 586.715101] alg: No test for adler32 (adler32-zlib) [ 587.598373] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 587.702666] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 589.303177] Key type lgssc registered [ 589.906549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 593.932218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 593.942723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.217253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.336935] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 601.789019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 602.004936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 603.946211] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 606.762335] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 608.613201] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 612.041440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 612.260025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.054249] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 618.216312] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 619.840877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 629.216862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 629.222242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.231650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 630.241740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 630.250311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.251813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 630.258484] Lustre: Skipped 1 previous similar message [ 634.739230] LustreError: 35545:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 634.816019] Lustre: server umount lustre-OST0000 complete [ 637.920763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 637.925577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.945502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.482561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 640.483539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.600360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.605330] Lustre: Skipped 2 previous similar messages [ 651.231133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 651.294364] LustreError: 35748:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 651.304069] LustreError: 35748:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 651.352302] Lustre: server umount lustre-MDT0000 complete [ 653.372754] LustreError: 35547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235507 with bad export cookie 12466522339509567942 [ 653.376393] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.379547] LustreError: 35547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 653.469935] LustreError: 35951:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 653.477359] LustreError: 35951:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 653.639639] Lustre: server umount lustre-MDT0001 complete [ 658.767847] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 660.214248] Key type lgssc unregistered [ 660.379823] LNet: 36482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.384500] LNetError: 36482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.394576] LNet: Removed LNI 192.168.202.134@tcp [ 660.852869] Key type .llcrypt unregistered [ 660.854591] Key type ._llcrypt unregistered [ 672.423192] Key type ._llcrypt registered [ 672.425913] Key type .llcrypt registered [ 672.482870] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 672.993435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 673.048787] alg: No test for adler32 (adler32-zlib) [ 673.988175] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 674.140350] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 675.751340] Key type lgssc registered [ 676.356626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 680.345200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 680.353923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.604340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.492640] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 687.690535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 687.859340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 689.970581] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 692.766798] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 694.425824] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 697.714823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 697.895137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.579495] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 703.118120] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 707.117954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 714.208274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 714.211987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.220875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 714.982041] Lustre: lustre-OST0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 717.283191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 717.285235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 717.285725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.517072] LustreError: 40104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 718.581713] Lustre: server umount lustre-OST0000 complete [ 732.534054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.542786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 732.545362] Lustre: Skipped 2 previous similar messages [ 737.760105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 737.763595] Lustre: Skipped 1 previous similar message [ 738.696819] LustreError: 40478:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 738.700989] LustreError: 40478:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 738.742933] Lustre: server umount lustre-MDT0000 complete [ 740.415336] LustreError: 37836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235594 with bad export cookie 8347735359957602831 [ 740.420624] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 740.425855] LustreError: 37836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 746.569043] LustreError: 40682:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 746.571720] LustreError: 40682:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 746.666294] Lustre: server umount lustre-MDT0001 complete [ 751.570276] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 752.993814] Key type lgssc unregistered [ 753.166463] LNet: 41213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.171162] LNetError: 41213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.182755] LNet: Removed LNI 192.168.202.134@tcp [ 753.610997] Key type .llcrypt unregistered [ 753.614554] Key type ._llcrypt unregistered [ 766.465316] Key type ._llcrypt registered [ 766.467337] Key type .llcrypt registered [ 766.523449] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 767.029461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 767.094613] alg: No test for adler32 (adler32-zlib) [ 768.005593] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 768.138721] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 769.775499] Key type lgssc registered [ 770.312064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 774.094863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 774.104965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.354205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.503506] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 782.248796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 782.450905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 784.234893] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 786.758322] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 788.326848] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 791.855695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 792.048717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.832723] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 797.771482] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 801.259941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 807.909960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 807.917359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.924904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 809.709361] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.34@tcp (stopping) [ 811.489423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 811.491361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.380320] LustreError: 44840:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 813.416097] Lustre: server umount lustre-MDT0000 complete [ 814.812727] LustreError: 43668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 814.825276] LustreError: 43668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 815.147104] LustreError: 44842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773235668 with bad export cookie 6301256463725202525 [ 815.152086] LustreError: 44842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 815.152788] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 816.608210] LustreError: 43668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.610722] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.625084] Lustre: Skipped 1 previous similar message [ 816.627927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 816.632112] Lustre: Skipped 2 previous similar messages [ 819.936517] LustreError: 43668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.640442] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 820.644328] Lustre: Skipped 1 previous similar message [ 824.742435] LustreError: 42590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.748859] LustreError: 42590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 829.407191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 829.458578] LustreError: 45043:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 829.461559] LustreError: 45043:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 829.543360] Lustre: server umount lustre-MDT0001 complete [ 842.157985] LustreError: 45268:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 842.162618] LustreError: 45268:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 842.225523] Lustre: server umount lustre-OST0000 complete [ 849.811992] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 851.415302] Key type lgssc unregistered [ 851.609351] LNet: 46103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.613906] LNetError: 46103:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.633726] LNet: Removed LNI 192.168.202.134@tcp [ 852.083914] Key type .llcrypt unregistered [ 852.085732] Key type ._llcrypt unregistered [ 864.656430] Key type ._llcrypt registered [ 864.657816] Key type .llcrypt registered [ 864.699557] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 865.185496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.201785] alg: No test for adler32 (adler32-zlib) [ 866.116340] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 866.227721] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 867.831186] Key type lgssc registered [ 868.349641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 872.239033] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 872.249375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 887.712861] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 892.831232] LustreError: 47449:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 919.519206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.540078] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 925.664454] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1091.331842] LustreError: 48158:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1091.378664] Lustre: server umount lustre-OST0000 complete [ 1098.434749] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1099.604884] Key type lgssc unregistered [ 1099.747438] LNet: 48995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.750765] LNetError: 48995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.759349] LNet: Removed LNI 192.168.202.134@tcp [ 1100.075988] Key type .llcrypt unregistered [ 1100.077936] Key type ._llcrypt unregistered [ 1111.349542] Key type ._llcrypt registered [ 1111.351574] Key type .llcrypt registered [ 1111.402116] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1111.869992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.893531] alg: No test for adler32 (adler32-zlib) [ 1112.790901] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1112.927318] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1114.527170] Key type lgssc registered [ 1115.141185] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1118.949727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1118.961107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.204085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.991807] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1125.882200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1126.047421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1127.835706] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1130.356817] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1132.218116] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1135.514649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1135.687949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.313569] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1140.797239] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1142.241427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.242676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.247703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1143.803066] LustreError: 52608:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1143.844741] Lustre: server umount lustre-OST0000 complete [ 1146.848223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1146.854829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.862890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.432708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.432857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.445299] Lustre: Skipped 1 previous similar message [ 1151.552132] LustreError: 52808:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1151.554077] LustreError: 52808:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1151.590267] Lustre: server umount lustre-MDT0000 complete [ 1153.223209] LustreError: 51084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236006 with bad export cookie 14958495690443806223 [ 1153.227462] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1153.227896] LustreError: 51084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1153.302744] LustreError: 53010:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1153.305714] LustreError: 53010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1153.373832] Lustre: server umount lustre-MDT0001 complete [ 1157.768363] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1159.175236] Key type lgssc unregistered [ 1159.324449] LNet: 53541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.327793] LNetError: 53541:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.336405] LNet: Removed LNI 192.168.202.134@tcp [ 1159.686797] Key type .llcrypt unregistered [ 1159.688571] Key type ._llcrypt unregistered [ 1170.441715] Key type ._llcrypt registered [ 1170.442888] Key type .llcrypt registered [ 1170.490335] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1170.858425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.904493] alg: No test for adler32 (adler32-zlib) [ 1171.781900] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1171.881942] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1173.471156] Key type lgssc registered [ 1173.872678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1177.031220] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1177.043803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1192.545147] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1197.663193] LustreError: 54884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1224.287173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.594659] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1229.000502] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1231.946260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.183574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.215942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1233.917613] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1237.477551] LustreError: 55830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.486649] LustreError: 55830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1237.659741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1237.816777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1239.433909] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1241.714363] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1243.306833] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1247.711972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1247.716543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.725153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1248.225299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1248.228076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.233220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1249.250132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1249.254604] Lustre: Skipped 1 previous similar message [ 1250.289101] LustreError: 57163:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1250.339481] Lustre: server umount lustre-OST0000 complete [ 1265.024790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.025629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1265.031242] Lustre: Skipped 1 previous similar message [ 1270.240821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1270.243778] Lustre: Skipped 2 previous similar messages [ 1271.166817] LustreError: 57541:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1271.171260] LustreError: 57541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1271.204036] Lustre: server umount lustre-MDT0000 complete [ 1272.872878] LustreError: 54892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236126 with bad export cookie 9849398304751565380 [ 1272.876953] LustreError: 54892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1272.877663] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.135720] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1287.199205] LustreError: 57742:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1287.208356] LustreError: 57742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1287.294542] Lustre: server umount lustre-MDT0001 complete [ 1291.859589] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1293.142256] Key type lgssc unregistered [ 1293.312459] LNet: 58275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.315375] LNetError: 58275:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.323434] LNet: Removed LNI 192.168.202.134@tcp [ 1293.662770] Key type .llcrypt unregistered [ 1293.663917] Key type ._llcrypt unregistered [ 1304.764252] Key type ._llcrypt registered [ 1304.765924] Key type .llcrypt registered [ 1304.809753] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1305.154851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.160489] alg: No test for adler32 (adler32-zlib) [ 1306.082437] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1306.203944] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1307.815166] Key type lgssc registered [ 1308.310567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1311.811915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1311.821132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.043251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.609956] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1318.060703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1318.235247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1319.887406] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1322.106323] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1323.642108] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1326.471487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1326.618149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.911731] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1331.078883] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1337.891842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1348.064745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1348.067463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.068420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1348.071790] Lustre: Skipped 1 previous similar message [ 1350.895064] LustreError: 61910:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1350.947638] Lustre: server umount lustre-OST0000 complete [ 1353.184826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.191219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1353.195417] Lustre: Skipped 2 previous similar messages [ 1358.304083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1358.306905] Lustre: Skipped 1 previous similar message [ 1358.528738] LustreError: 62112:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1358.531760] LustreError: 62112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1358.560421] Lustre: server umount lustre-MDT0000 complete [ 1359.946412] LustreError: 59626:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236213 with bad export cookie 11073949510782844354 [ 1359.949559] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.950590] LustreError: 59626:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1360.015543] LustreError: 62313:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1360.018012] LustreError: 62313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1360.073747] Lustre: server umount lustre-MDT0001 complete [ 1364.294335] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1365.431922] Key type lgssc unregistered [ 1365.562521] LNet: 62844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.565511] LNetError: 62844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.574750] LNet: Removed LNI 192.168.202.134@tcp [ 1365.901334] Key type .llcrypt unregistered [ 1365.902657] Key type ._llcrypt unregistered [ 1379.128298] Key type ._llcrypt registered [ 1379.129952] Key type .llcrypt registered [ 1379.175752] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1379.763399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.776562] alg: No test for adler32 (adler32-zlib) [ 1380.653948] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1380.751825] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1382.343209] Key type lgssc registered [ 1382.756108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1385.707387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1385.720892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.922433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.376409] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1391.500735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1391.625947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.022216] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1394.916964] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1396.180373] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1398.526531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1398.633987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.517611] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1402.386910] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1412.962957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1420.345552] Lustre: Failing over lustre-MDT0000 [ 1420.472942] LustreError: 66789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1420.518972] Lustre: server umount lustre-MDT0000 complete [ 1422.305488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.308872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.314473] LustreError: 65235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.320620] LustreError: 65235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1423.081417] LustreError: 64547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.329497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.333992] Lustre: Skipped 1 previous similar message [ 1424.726423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.793430] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.852714] Lustre: *** cfs_fail_loc=135, val=0*** [ 1424.898623] LustreError: 67298:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1424.903327] LustreError: 67298:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -2 [ 1424.907502] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1424.907502] [ 1424.911798] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1424.919038] LustreError: 67286:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1424.923388] LustreError: 67286:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1424.929151] LustreError: 67286:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1424.940236] Lustre: server umount lustre-MDT0000 complete [ 1424.942445] LustreError: 67286:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1427.643504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.697298] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1427.796800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.817182] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1428.193491] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1429.392783] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1433.065307] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1433.071608] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1433.088607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1437.663968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.666722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.671711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1438.179133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1438.182052] Lustre: Skipped 1 previous similar message [ 1440.173902] LustreError: 68106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1440.176598] LustreError: 68106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1440.215695] Lustre: server umount lustre-OST0000 complete [ 1443.296978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.297139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1443.298071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.301938] Lustre: Skipped 2 previous similar messages [ 1443.305080] LustreError: Skipped 1 previous similar message [ 1447.676580] LustreError: 68308:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1447.678515] LustreError: 68308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1447.710247] Lustre: server umount lustre-MDT0000 complete [ 1448.420224] LustreError: 65235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.426560] LustreError: 65235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1449.127357] LustreError: 64503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236302 with bad export cookie 11169915899972965061 [ 1449.130557] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1449.132632] LustreError: 64503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1449.249313] Lustre: server umount lustre-MDT0001 complete [ 1453.406879] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1454.519457] Key type lgssc unregistered [ 1454.647553] LNet: 69041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.650443] LNetError: 69041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.659332] LNet: Removed LNI 192.168.202.134@tcp [ 1454.956083] Key type .llcrypt unregistered [ 1454.957213] Key type ._llcrypt unregistered [ 1464.649112] Key type ._llcrypt registered [ 1464.650403] Key type .llcrypt registered [ 1464.694477] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1465.079130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1465.132081] alg: No test for adler32 (adler32-zlib) [ 1465.998850] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1466.090436] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1467.679165] Key type lgssc registered [ 1468.094652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.139574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.148556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.374435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.854124] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1477.087646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1477.252614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1478.834535] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1480.895462] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.339723] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1485.040190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1485.183747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.485242] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1489.509116] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.510667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1502.243237] Lustre: Failing over lustre-MDT0000 [ 1502.357530] LustreError: 72677:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1502.385144] Lustre: server umount lustre-MDT0000 complete [ 1502.688970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.690301] LustreError: 70407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.694259] Lustre: Skipped 1 previous similar message [ 1502.700437] LustreError: 70407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1504.997055] LustreError: 70406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.245888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.313873] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.379884] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1506.444085] LustreError: 73186:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1506.449066] LustreError: 73186:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -9 [ 1506.453052] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1506.453052] [ 1506.457580] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1506.465543] LustreError: 73174:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1506.468990] LustreError: 73174:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1506.473172] LustreError: 73174:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1506.488589] Lustre: server umount lustre-MDT0000 complete [ 1506.490745] LustreError: 73174:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1507.808633] LustreError: 70412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.814970] LustreError: 70412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1509.017176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.072054] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.169691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.189264] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1510.112458] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1510.695315] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1514.476461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1514.482711] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1514.503371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1518.047895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1518.051689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.056597] Lustre: Skipped 1 previous similar message [ 1518.059083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1519.584675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1519.588078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.588497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1519.595865] Lustre: Skipped 1 previous similar message [ 1521.655138] LustreError: 73997:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1521.658441] LustreError: 73997:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1521.709382] Lustre: server umount lustre-OST0000 complete [ 1524.705384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.706815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1524.712562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1524.717819] Lustre: Skipped 1 previous similar message [ 1529.341699] LustreError: 74199:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1529.343918] LustreError: 74199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1529.378228] Lustre: server umount lustre-MDT0000 complete [ 1529.824387] LustreError: 70407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.833331] LustreError: 70407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1530.918039] LustreError: 70391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236384 with bad export cookie 6680578739591814471 [ 1530.924887] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.068773] Lustre: server umount lustre-MDT0001 complete [ 1535.329224] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1536.489945] Key type lgssc unregistered [ 1536.624419] LNet: 74932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.628091] LNetError: 74932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.637498] LNet: Removed LNI 192.168.202.134@tcp [ 1536.976663] Key type .llcrypt unregistered [ 1536.978236] Key type ._llcrypt unregistered [ 1541.716288] Key type ._llcrypt registered [ 1541.717947] Key type .llcrypt registered [ 1541.755323] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1541.861028] Key type .llcrypt unregistered [ 1541.862723] Key type ._llcrypt unregistered [ 1551.628783] Key type ._llcrypt registered [ 1551.630332] Key type .llcrypt registered [ 1551.674658] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1552.121153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.143887] alg: No test for adler32 (adler32-zlib) [ 1553.012254] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1553.097297] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1554.688294] Key type lgssc registered [ 1555.079880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1558.110426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1558.118857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.345737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.014342] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1564.671328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1564.848273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.347848] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1568.793888] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1570.240344] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.045550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.173913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1575.450126] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1577.457633] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1584.423141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1594.850991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1594.851272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.852918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.860516] Lustre: Skipped 1 previous similar message [ 1596.912666] LustreError: 79037:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1596.962560] Lustre: server umount lustre-OST0000 complete [ 1599.968943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.976672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.979601] Lustre: Skipped 2 previous similar messages [ 1604.676040] LustreError: 79239:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1604.678919] LustreError: 79239:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1604.706239] Lustre: server umount lustre-MDT0000 complete [ 1605.088357] LustreError: 76771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.094941] LustreError: 76771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1606.257423] LustreError: 78938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236459 with bad export cookie 9002437020234640682 [ 1606.259795] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.260833] LustreError: 78938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1606.341408] LustreError: 79440:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1606.344465] LustreError: 79440:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1606.407321] Lustre: server umount lustre-MDT0001 complete [ 1610.320173] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1611.483490] Key type lgssc unregistered [ 1611.632366] LNet: 79971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.635513] LNetError: 79971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.642344] LNet: Removed LNI 192.168.202.134@tcp [ 1611.971502] Key type .llcrypt unregistered [ 1611.973063] Key type ._llcrypt unregistered [ 1621.385198] Key type ._llcrypt registered [ 1621.386374] Key type .llcrypt registered [ 1621.426484] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1621.819396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1621.839919] alg: No test for adler32 (adler32-zlib) [ 1622.696353] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1622.775601] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1624.359139] Key type lgssc registered [ 1624.801934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1628.049841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.058437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.246092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.869920] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1634.300940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1634.464789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1635.927614] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1638.350437] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1639.940264] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1643.457891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1643.642068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.131326] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1648.653798] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1653.866716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1659.362453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.367612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.368534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.317705] LustreError: 83586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1662.356337] Lustre: server umount lustre-OST0000 complete [ 1664.480875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.481539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.486266] Lustre: Skipped 2 previous similar messages [ 1664.488640] Lustre: Skipped 1 previous similar message [ 1669.600988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1669.605657] Lustre: Skipped 1 previous similar message [ 1674.720053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.722183] Lustre: Skipped 1 previous similar message [ 1678.303149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1678.337443] LustreError: 83787:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1678.340154] LustreError: 83787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1678.370709] Lustre: server umount lustre-MDT0000 complete [ 1679.840670] LustreError: 82502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1679.846771] LustreError: 82502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1679.890271] LustreError: 81320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236533 with bad export cookie 12063906797106960741 [ 1679.894348] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.896326] LustreError: 81320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1679.977449] LustreError: 83991:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1679.980203] LustreError: 83991:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1680.034226] Lustre: server umount lustre-MDT0001 complete [ 1684.513568] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1685.746468] Key type lgssc unregistered [ 1685.898435] LNet: 84521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.901498] LNetError: 84521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.909423] LNet: Removed LNI 192.168.202.134@tcp [ 1686.269215] Key type .llcrypt unregistered [ 1686.270634] Key type ._llcrypt unregistered [ 1696.389204] Key type ._llcrypt registered [ 1696.390884] Key type .llcrypt registered [ 1696.443458] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1696.909886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.981074] alg: No test for adler32 (adler32-zlib) [ 1697.897687] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1698.003777] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1699.599182] Key type lgssc registered [ 1700.132667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.266210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.276309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.509598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.198281] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1710.845162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1711.016058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1712.632561] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1714.988941] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1716.537593] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1719.400849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1719.563199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.029692] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1724.635716] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1730.857612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1741.281076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1741.283477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.284146] LustreError: Skipped 1 previous similar message [ 1741.289196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.292783] Lustre: Skipped 1 previous similar message [ 1744.814073] LustreError: 88163:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1744.852646] Lustre: server umount lustre-OST0000 complete [ 1746.403494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1746.411213] Lustre: Skipped 1 previous similar message [ 1746.413716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1746.415750] Lustre: Skipped 1 previous similar message [ 1751.520151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.522360] Lustre: Skipped 1 previous similar message [ 1752.519583] LustreError: 88365:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1752.522500] LustreError: 88365:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1752.554749] Lustre: server umount lustre-MDT0000 complete [ 1754.070344] LustreError: 85873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236607 with bad export cookie 16006105851346668080 [ 1754.073156] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1754.075068] LustreError: 85873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1754.143350] LustreError: 88566:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1754.146420] LustreError: 88566:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1754.198220] Lustre: server umount lustre-MDT0001 complete [ 1758.245961] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1759.341196] Key type lgssc unregistered [ 1759.481552] LNet: 89098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.484267] LNetError: 89098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.492485] LNet: Removed LNI 192.168.202.134@tcp [ 1759.812054] Key type .llcrypt unregistered [ 1759.814372] Key type ._llcrypt unregistered [ 1769.393293] Key type ._llcrypt registered [ 1769.394867] Key type .llcrypt registered [ 1769.443937] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1769.766699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.777837] alg: No test for adler32 (adler32-zlib) [ 1770.650602] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1770.745267] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1772.327155] Key type lgssc registered [ 1772.761297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.825409] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.836931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1791.327302] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1796.447197] LustreError: 90442:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 1823.071218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.237628] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1827.279049] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1829.436846] LustreError: 91298:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1829.471569] Lustre: server umount lustre-OST0000 complete [ 1832.415168] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:44:45 (1773236685) [ 1834.866802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.031081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.552911] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1839.724217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1839.870078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.436533] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1843.601454] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1845.034777] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1847.921254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1848.073433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.291872] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1852.279357] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1859.370374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1869.794815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.795237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1869.796744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.800685] Lustre: Skipped 1 previous similar message [ 1872.691091] LustreError: 94392:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1872.693898] LustreError: 94392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1872.735385] Lustre: server umount lustre-OST0000 complete [ 1874.912474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.923248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.925721] Lustre: Skipped 2 previous similar messages [ 1880.032224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.035338] Lustre: Skipped 1 previous similar message [ 1880.510818] LustreError: 94594:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1880.513397] LustreError: 94594:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1880.539880] Lustre: server umount lustre-MDT0000 complete [ 1882.029257] LustreError: 91947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236735 with bad export cookie 11051137055378323638 [ 1882.032358] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.033725] LustreError: 91947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1882.154660] Lustre: server umount lustre-MDT0001 complete [ 1886.297038] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1887.550259] Key type lgssc unregistered [ 1887.707938] LNet: 95327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.713480] LNetError: 95327:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.731157] LNet: Removed LNI 192.168.202.134@tcp [ 1888.086467] Key type .llcrypt unregistered [ 1888.087833] Key type ._llcrypt unregistered [ 1898.655978] Key type ._llcrypt registered [ 1898.657623] Key type .llcrypt registered [ 1898.706130] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1899.103060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.142921] alg: No test for adler32 (adler32-zlib) [ 1900.038584] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1900.151628] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1901.743137] Key type lgssc registered [ 1902.171532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1905.303191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1905.328782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.558657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.172554] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1911.607235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1911.760347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.209845] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1915.199637] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1916.568253] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1919.120047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1919.265225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.588517] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 1923.645199] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1931.559217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1941.985145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.986771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1941.990475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.992951] LustreError: Skipped 1 previous similar message [ 1941.995068] Lustre: Skipped 1 previous similar message [ 1947.105269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.107852] Lustre: Skipped 1 previous similar message [ 1951.199140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1951.233166] LustreError: 98961:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1951.280618] Lustre: server umount lustre-OST0000 complete [ 1957.347239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.351972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.355845] Lustre: Skipped 2 previous similar messages [ 1957.361626] Lustre: Skipped 1 previous similar message [ 1962.464956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.467557] Lustre: Skipped 1 previous similar message [ 1967.071213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1967.113207] LustreError: 99166:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1967.115203] LustreError: 99166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1967.140249] Lustre: server umount lustre-MDT0000 complete [ 1967.583953] LustreError: 96696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.589295] LustreError: 96696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1968.466101] LustreError: 98862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236822 with bad export cookie 7033268907730024812 [ 1968.470161] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.470918] LustreError: 98862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1968.539099] LustreError: 99368:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1968.541877] LustreError: 99368:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1968.592537] Lustre: server umount lustre-MDT0001 complete [ 1972.414485] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 1973.521851] Key type lgssc unregistered [ 1973.660363] LNet: 99899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.663421] LNetError: 99899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.676318] LNet: Removed LNI 192.168.202.134@tcp [ 1973.981485] Key type .llcrypt unregistered [ 1973.982983] Key type ._llcrypt unregistered [ 1981.798160] Key type ._llcrypt registered [ 1981.799618] Key type .llcrypt registered [ 1981.842504] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 1982.156763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1982.168146] alg: No test for adler32 (adler32-zlib) [ 1983.054168] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 1983.151466] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 1984.751213] Key type lgssc registered [ 1985.237744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1988.355079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1988.366813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2003.807332] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2008.927189] LustreError: 101098:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2035.551240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.562377] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2039.580234] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2042.138553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.243773] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2042.249541] LustreError: 102009:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2042.253027] LustreError: 102009:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2042.255885] LustreError: 102009:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2042.295487] Lustre: server umount lustre-MDT0000 complete [ 2042.297732] LustreError: 102009:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2055.647775] LustreError: 102441:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2055.716592] Lustre: server umount lustre-OST0000 complete [ 2059.809662] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2063.557579] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2067.417803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2069.933392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2072.243032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2074.621616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2077.338718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2077.362583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.451717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2077.466996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2077.504534] Lustre: lustre-MDT0000: new disk, initializing [ 2077.535125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.543781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2079.072602] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2084.049621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2084.080406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2084.125150] Lustre: 104776:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2084.150894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2084.152794] Lustre: Skipped 1 previous similar message [ 2084.187164] Lustre: lustre-MDT0001: new disk, initializing [ 2084.223028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.242180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2084.247289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2085.782151] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2089.297688] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2090.862744] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2093.501504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2093.534512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2093.640252] Lustre: lustre-OST0000: new disk, initializing [ 2093.642842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2093.678278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.844091] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2099.455432] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2100.229900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2100.234093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2100.266395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2101.037387] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2101.123315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2102.591615] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2102.669170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2105.312860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2105.316129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.316692] LustreError: Skipped 1 previous similar message [ 2105.322197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2105.328397] Lustre: Skipped 1 previous similar message [ 2109.487184] LustreError: 107713:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2109.490626] LustreError: 107713:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2109.536046] Lustre: server umount lustre-OST0000 complete [ 2115.040061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2115.043725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.048284] Lustre: Skipped 1 previous similar message [ 2115.050417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.377681] LustreError: 107914:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2117.380062] LustreError: 107914:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2117.417637] Lustre: server umount lustre-MDT0000 complete [ 2118.915267] LustreError: 104767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773236972 with bad export cookie 7752087085938488668 [ 2118.918681] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2118.920388] LustreError: 104767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2119.047815] Lustre: server umount lustre-MDT0001 complete [ 2122.049834] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:49:35 (1773236975) [ 2123.200459] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2128.922972] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2132.099251] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2135.566057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2138.042076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2140.427340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2142.803050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2145.206924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2145.233598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.328436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2145.344136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2145.380190] Lustre: lustre-MDT0000: new disk, initializing [ 2145.406990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.414146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2146.886880] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2151.368860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2151.401283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2151.437828] Lustre: 111244:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2151.461925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2151.464153] Lustre: Skipped 1 previous similar message [ 2151.502247] Lustre: lustre-MDT0001: new disk, initializing [ 2151.547656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2151.551361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2152.951708] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2156.030242] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2157.234231] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2159.350389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2159.375203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2159.469566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2159.500394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.503289] Lustre: Skipped 1 previous similar message [ 2161.381965] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2164.590617] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2166.071599] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2166.773756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2166.777042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2166.810088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2167.183736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2168.571367] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2168.660894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2171.873739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2171.874305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.875411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.875414] Lustre: Skipped 1 previous similar message [ 2171.876802] LustreError: Skipped 1 previous similar message [ 2171.881724] Lustre: Skipped 2 previous similar messages [ 2175.470194] LustreError: 114186:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2175.472841] LustreError: 114186:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2175.525941] Lustre: server umount lustre-OST0000 complete [ 2176.992195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.993203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.995939] Lustre: Skipped 1 previous similar message [ 2176.997581] Lustre: Skipped 1 previous similar message [ 2182.112085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2182.114021] Lustre: Skipped 1 previous similar message [ 2191.327108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2191.328549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.333147] Lustre: Skipped 3 previous similar messages [ 2191.364849] LustreError: 114387:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2191.366940] LustreError: 114387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2191.395353] Lustre: server umount lustre-MDT0000 complete [ 2192.706474] LustreError: 114389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237046 with bad export cookie 7752087085938489536 [ 2192.710459] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.711319] LustreError: 114389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2192.834293] Lustre: server umount lustre-MDT0001 complete [ 2195.712236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.903184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.151421] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2200.005040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2201.428474] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2203.298902] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2204.623812] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2207.022705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2209.003273] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2210.959682] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2223.804334] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2225.635533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.635818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2225.636061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2225.640644] Lustre: Skipped 1 previous similar message [ 2226.860987] LustreError: 117429:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2226.863027] LustreError: 117429:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2226.894873] Lustre: server umount lustre-OST0000 complete [ 2230.300501] Lustre: server umount lustre-MDT0000 complete [ 2230.753087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2230.755362] LustreError: 115084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2230.755961] LustreError: Skipped 1 previous similar message [ 2230.763078] LustreError: 115084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2231.717191] LustreError: 115048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237085 with bad export cookie 7752087085938490418 [ 2231.720618] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2231.725710] LustreError: 115048:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2231.855757] Lustre: server umount lustre-MDT0001 complete [ 2235.453550] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2236.549126] Key type lgssc unregistered [ 2236.679383] LNet: 118360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.682291] LNetError: 118360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.690335] LNet: Removed LNI 192.168.202.134@tcp [ 2237.035760] Key type .llcrypt unregistered [ 2237.037277] Key type ._llcrypt unregistered [ 2243.192386] Key type ._llcrypt registered [ 2243.195153] Key type .llcrypt registered [ 2243.237233] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 2249.177980] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2249.472436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.487337] alg: No test for adler32 (adler32-zlib) [ 2250.346764] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2250.428026] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2252.007110] Key type lgssc registered [ 2252.349043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2255.241128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2257.598901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2259.827820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2262.078534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2264.454547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2264.474182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2264.479557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.569285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2265.590739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2265.628922] Lustre: lustre-MDT0000: new disk, initializing [ 2265.660816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.668561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2266.966162] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2271.346295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.376242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2271.413360] Lustre: 121460:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2271.440289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2271.442255] Lustre: Skipped 1 previous similar message [ 2271.484101] Lustre: lustre-MDT0001: new disk, initializing [ 2271.519321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2271.532160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2271.535807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2272.973758] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2276.130081] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2277.343279] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.613491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2279.640683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2279.748332] Lustre: lustre-OST0000: new disk, initializing [ 2279.750757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2279.782913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.750985] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2285.121701] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2286.683171] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2289.660048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2289.663498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2289.695643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2289.837913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2291.188644] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2291.266237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2294.756662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.757701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2294.758113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.760914] Lustre: Skipped 1 previous similar message [ 2298.031346] LustreError: 124418:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2298.071572] Lustre: server umount lustre-OST0000 complete [ 2299.872471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.877934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.880240] Lustre: Skipped 2 previous similar messages [ 2304.992450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.001666] Lustre: Skipped 1 previous similar message [ 2305.471598] LustreError: 124620:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2305.474017] LustreError: 124620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2305.503820] Lustre: server umount lustre-MDT0000 complete [ 2306.739129] LustreError: 122240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237160 with bad export cookie 15470509837072366207 [ 2306.741852] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.743720] LustreError: 122240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2306.808349] LustreError: 124822:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2306.810926] LustreError: 124822:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2306.865133] Lustre: server umount lustre-MDT0001 complete [ 2309.705345] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 09:52:43 (1773237163) [ 2312.207199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.401734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.763361] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2316.786303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2316.904276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2318.252149] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2320.203872] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2321.509898] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2322.239588] LustreError: 125522:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2322.243923] LustreError: 125522:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2322.247170] LustreError: 125522:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2322.401036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.402049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.408294] Lustre: Skipped 2 previous similar messages [ 2326.496246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2326.498866] Lustre: Skipped 2 previous similar messages [ 2328.386307] LustreError: 126833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2328.388652] LustreError: 126833:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2328.421856] Lustre: server umount lustre-MDT0000 complete [ 2329.819330] LustreError: 125472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237183 with bad export cookie 15470509837072367075 [ 2329.821216] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2329.825106] LustreError: 125472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2329.938162] Lustre: server umount lustre-MDT0001 complete [ 2332.952643] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 09:53:06 (1773237186) [ 2335.497184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2350.943646] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2356.064082] LustreError: 127679:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2381.663473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.603641] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2385.475256] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2386.224465] LustreError: 128345:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2386.227130] LustreError: 128345:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2386.264337] Lustre: server umount lustre-OST0000 complete [ 2389.314932] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 09:54:02 (1773237242) [ 2391.739870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.898321] LustreError: 129016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.906730] LustreError: 129016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2393.294802] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2396.406610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2397.938182] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2399.839628] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2401.198568] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2403.537561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2403.652852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.656307] Lustre: Skipped 2 previous similar messages [ 2405.506538] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2407.235730] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2422.242092] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.242106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.243645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.249575] Lustre: Skipped 1 previous similar message [ 2426.050101] LustreError: 131284:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2426.052146] LustreError: 131284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2426.084673] Lustre: server umount lustre-MDT0000 complete [ 2427.361950] LustreError: 129015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.367881] LustreError: 129015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2427.404163] LustreError: 130635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237281 with bad export cookie 15470509837072367698 [ 2427.407556] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.408643] LustreError: 130635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2427.520892] Lustre: server umount lustre-MDT0001 complete [ 2438.999734] Lustre: server umount lustre-OST0000 complete [ 2441.690317] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 09:54:55 (1773237295) [ 2443.855691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.998861] LustreError: 132357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.023573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.329279] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2448.153567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2449.546741] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2451.688620] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2452.876610] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2455.050800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2457.109093] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2458.889771] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2460.219764] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2468.520750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2469.495737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2470.889218] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2470.965517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2473.952812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2473.956420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.960421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.962636] Lustre: Skipped 3 previous similar messages [ 2477.677072] LustreError: 135162:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2477.679942] LustreError: 135162:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2477.717151] Lustre: server umount lustre-OST0000 complete [ 2479.072690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2479.075891] LustreError: Skipped 1 previous similar message [ 2485.076820] Lustre: server umount lustre-MDT0000 complete [ 2486.308985] LustreError: 133977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237340 with bad export cookie 15470509837072368762 [ 2486.311528] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2486.312430] LustreError: 133977:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2486.422856] Lustre: server umount lustre-MDT0001 complete [ 2489.089141] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 09:55:42 (1773237342) [ 2491.314201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2506.783815] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2511.903309] LustreError: 136209:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2537.503282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.506215] Lustre: Skipped 2 previous similar messages [ 2539.283604] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2541.021573] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2543.228629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.375437] LustreError: 136236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.381422] LustreError: 136236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2543.415642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2544.575739] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2547.215858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2548.455601] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2549.985275] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2551.121065] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2552.519807] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2552.603466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2553.959683] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2554.045787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2556.897726] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.899155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.903667] Lustre: Skipped 4 previous similar messages [ 2556.906791] Lustre: Skipped 4 previous similar messages [ 2560.839323] LustreError: 138984:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2560.842396] LustreError: 138984:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2560.873308] Lustre: server umount lustre-MDT0000 complete [ 2562.017585] LustreError: 138987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.023876] LustreError: 138987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2562.308170] LustreError: 137162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237416 with bad export cookie 15470509837072369770 [ 2562.311101] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.312672] LustreError: 137162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2562.424715] Lustre: server umount lustre-MDT0001 complete [ 2573.978318] Lustre: server umount lustre-OST0000 complete [ 2576.851818] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 09:57:10 (1773237430) [ 2579.033192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2594.463524] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2599.583469] LustreError: 140031:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 2627.020284] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2628.738866] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.114705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.259836] LustreError: 140058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.303047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2632.693388] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2635.754809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2637.377924] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2639.210598] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2640.447786] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2642.886786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2642.918356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2642.978609] Lustre: lustre-OST0001: new disk, initializing [ 2642.981052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2644.980551] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2645.303305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2645.307286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2645.354548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2648.294979] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2649.894600] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2649.991562] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2651.630356] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2651.727047] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2655.714830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.717532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.719668] Lustre: Skipped 2 previous similar messages [ 2655.723435] Lustre: Skipped 1 previous similar message [ 2658.602849] LustreError: 143763:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2658.604979] LustreError: 143763:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2658.624657] Lustre: server umount lustre-OST0000 complete [ 2660.838149] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.838963] LustreError: 141673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.841061] LustreError: Skipped 1 previous similar message [ 2660.847335] LustreError: 141673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2666.134147] Lustre: server umount lustre-OST0001 complete [ 2673.692290] Lustre: server umount lustre-MDT0000 complete [ 2675.164136] LustreError: 140984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237528 with bad export cookie 15470509837072370715 [ 2675.168740] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.168983] LustreError: 140984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2675.312709] Lustre: server umount lustre-MDT0001 complete [ 2677.679218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2679.269138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2680.809959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2682.333175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2684.539327] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 09:58:58 (1773237538) [ 2685.083746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2685.616359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2686.176249] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 09:58:59 (1773237539) [ 2688.604073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.703261] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2688.714589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2688.785668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.788095] Lustre: Skipped 6 previous similar messages [ 2690.105520] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2693.010739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2693.043226] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2693.053655] Lustre: 146004:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2694.411985] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2696.079781] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.201802] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2699.384318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2699.447484] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2701.363843] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2703.072809] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2703.119859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2704.404418] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2704.481191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2705.766884] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2705.851976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2708.449208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2708.451555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.451652] LustreError: Skipped 1 previous similar message [ 2708.455358] Lustre: Skipped 4 previous similar messages [ 2712.597917] Lustre: server umount lustre-OST0000 complete [ 2726.372050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2728.288641] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2728.548230] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2728.552823] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2728.552870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2730.004814] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2731.446138] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2731.525786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2732.746344] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2732.818551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2733.937814] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94c2c72ec800.ost_server_uuid 50 [ 2734.462172] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff94c2c72ec800.ost_server_uuid in FULL state after 0 sec [ 2734.521687] LustreError: 146038:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.34@tcp) returned error from blocking AST (req@00000000c346f4db x1859373869748352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000003bf2e454/0xd6b24abb04e29560 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.34@tcp remote: 0x3890f7c0991d0689 expref: 5 pid: 146038 timeout: 2834 lvb_type: 0 [ 2734.534976] LustreError: lustre-MDT0000: A client on nid 192.168.202.34@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2734.539464] LustreError: 145999:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.34@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000003bf2e454/0xd6b24abb04e29560 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.34@tcp remote: 0x3890f7c0991d0689 expref: 6 pid: 146038 timeout: 0 lvb_type: 0 [ 2739.169187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2739.172271] LustreError: Skipped 1 previous similar message [ 2741.709868] Lustre: server umount lustre-OST0000 complete [ 2749.015419] Lustre: server umount lustre-MDT0000 complete [ 2749.407996] LustreError: 146014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.413467] LustreError: 146014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2750.236233] LustreError: 145994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237603 with bad export cookie 15470509837072371891 [ 2750.239898] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.240544] LustreError: 145994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2756.401247] Lustre: server umount lustre-MDT0001 complete [ 2760.015397] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2761.072087] Key type lgssc unregistered [ 2761.217709] LNet: 151486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2761.223264] LNetError: 151486:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2761.236486] LNet: Removed LNI 192.168.202.134@tcp [ 2761.561032] Key type .llcrypt unregistered [ 2761.562589] Key type ._llcrypt unregistered [ 2770.677766] Key type ._llcrypt registered [ 2770.679184] Key type .llcrypt registered [ 2770.720439] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2771.043037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2771.051695] alg: No test for adler32 (adler32-zlib) [ 2771.907631] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2771.991627] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2773.575154] Key type lgssc registered [ 2773.966627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2776.609242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2776.617066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.780332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.122671] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2782.033375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.185951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.493863] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2785.325612] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2786.491944] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2788.796709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2788.931457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.991280] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2792.974571] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2803.301745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2805.566494] Lustre: Failing over lustre-MDT0000 [ 2805.755862] LustreError: 155151:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2805.784083] Lustre: server umount lustre-MDT0000 complete [ 2807.776870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.779353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.783593] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.788435] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2808.800499] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.801739] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.808413] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2808.812039] Lustre: Skipped 1 previous similar message [ 2813.920990] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.928240] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2816.673466] LustreError: 155153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.34@tcp arrived at 1773237670 with bad export cookie 18322277945417779920 [ 2816.677959] LustreError: 155153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2819.040763] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.045990] LustreError: 152883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2819.058030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.098995] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.183936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.205232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2820.610449] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2821.415428] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2821.418435] Lustre: lustre-MDT0000: Denying connection for new client 663b14d7-9324-4f3a-995b-e86a6ae7577e (at 192.168.202.34@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2824.676961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2826.462418] Lustre: lustre-MDT0000: Denying connection for new client 663b14d7-9324-4f3a-995b-e86a6ae7577e (at 192.168.202.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 2826.468304] Lustre: Skipped 1 previous similar message [ 2828.256372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.259609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.266306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.793914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.796785] Lustre: Skipped 1 previous similar message [ 2834.284111] LustreError: 156098:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2834.285959] LustreError: 156098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2834.321879] Lustre: server umount lustre-OST0000 complete [ 2835.507672] LustreError: 156300:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2835.509944] Lustre: 155665:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2835.513447] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2835.520868] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2835.524911] LustreError: 155665:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000411ca2d7 x1859374415907456/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2835.531276] LustreError: 155665:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2835.534335] LustreError: 155665:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2835.537940] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2835.611724] LustreError: 156300:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2835.614924] LustreError: 156300:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2835.640465] Lustre: server umount lustre-MDT0000 complete [ 2836.851196] LustreError: 152864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237690 with bad export cookie 18322277945417780270 [ 2836.855382] LustreError: 152864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2836.855558] LustreError: 152193:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000026332e3e x1859374415908352/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2836.855968] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2843.005945] LustreError: 156501:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2843.008292] LustreError: 156501:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2843.065965] Lustre: server umount lustre-MDT0001 complete [ 2846.495566] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2847.507030] Key type lgssc unregistered [ 2847.631364] LNet: 157033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2847.634055] LNetError: 157033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2847.642302] LNet: Removed LNI 192.168.202.134@tcp [ 2847.944117] Key type .llcrypt unregistered [ 2847.945821] Key type ._llcrypt unregistered [ 2856.801368] Key type ._llcrypt registered [ 2856.802668] Key type .llcrypt registered [ 2856.838347] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2857.249527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2857.262097] alg: No test for adler32 (adler32-zlib) [ 2858.110813] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2858.191808] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2859.775114] Key type lgssc registered [ 2860.125494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2862.759984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2862.766049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2863.920100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.183039] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2868.159846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2868.291166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2869.578954] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2871.313755] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2872.451131] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2874.781471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2874.903980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.776634] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2878.557193] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2890.278207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2895.329661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.329976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.330738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.333764] Lustre: Skipped 1 previous similar message [ 2897.198065] LustreError: 160669:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2897.234117] Lustre: server umount lustre-OST0000 complete [ 2898.911904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2898.914828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.919429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.921141] Lustre: Skipped 1 previous similar message [ 2900.448216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2900.449114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.542550] LustreError: 160871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2900.545676] LustreError: 160871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2900.579162] Lustre: server umount lustre-MDT0000 complete [ 2901.840226] LustreError: 158384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237755 with bad export cookie 2931374870594266180 [ 2901.842926] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.844035] LustreError: 158384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2901.905445] LustreError: 161072:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2901.907839] LustreError: 161072:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2901.961452] Lustre: server umount lustre-MDT0001 complete [ 2905.284533] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2906.271144] Key type lgssc unregistered [ 2906.387384] LNet: 161604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2906.390457] LNetError: 161604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2906.399268] LNet: Removed LNI 192.168.202.134@tcp [ 2906.681406] Key type .llcrypt unregistered [ 2906.682415] Key type ._llcrypt unregistered [ 2917.996339] Key type ._llcrypt registered [ 2917.997756] Key type .llcrypt registered [ 2918.035932] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2918.453597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2918.465305] alg: No test for adler32 (adler32-zlib) [ 2919.323829] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2919.409296] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2920.991146] Key type lgssc registered [ 2921.362934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2924.132557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2924.139343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.319884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2926.680290] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2929.684821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2929.828225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.139622] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2932.756782] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2933.910673] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2936.056663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2936.171976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.190505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2938.085836] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2939.897295] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2942.431987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2942.435501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.441759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.506751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2945.507403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.510531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.270090] LustreError: 165507:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2948.305460] Lustre: server umount lustre-OST0000 complete [ 2950.625083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.625557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.625809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.625814] Lustre: Skipped 1 previous similar message [ 2950.629419] Lustre: Skipped 1 previous similar message [ 2955.652934] LustreError: 165709:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2955.655437] LustreError: 165709:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2955.687303] Lustre: server umount lustre-MDT0000 complete [ 2955.744406] LustreError: 163261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2955.750735] LustreError: 163261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2956.930405] LustreError: 163241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237810 with bad export cookie 17510054916289955315 [ 2956.932744] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2956.934986] LustreError: 163241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2957.008233] LustreError: 165910:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2957.010087] LustreError: 165910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2957.060948] Lustre: server umount lustre-MDT0001 complete [ 2960.532733] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 2961.620919] Key type lgssc unregistered [ 2961.754309] LNet: 166440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2961.757360] LNetError: 166440:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2961.767377] LNet: Removed LNI 192.168.202.134@tcp [ 2962.073867] Key type .llcrypt unregistered [ 2962.075121] Key type ._llcrypt unregistered [ 2970.096278] Key type ._llcrypt registered [ 2970.097530] Key type .llcrypt registered [ 2970.138713] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 2970.534499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2970.607927] alg: No test for adler32 (adler32-zlib) [ 2971.465964] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 2971.554747] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 2973.143161] Key type lgssc registered [ 2973.550965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2977.195251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2977.200983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.329263] Lustre: *** cfs_fail_loc=135, val=0*** [ 2978.372090] LustreError: 167923:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2978.374820] LustreError: 167923:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.134@tcp: cfg command failed: rc = -2 [ 2978.378226] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2978.378226] [ 2978.381863] LustreError: MGC192.168.202.134@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2978.387550] LustreError: 167883:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2978.390962] LustreError: 167883:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2978.394060] LustreError: 167883:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2978.397350] LustreError: 167883:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2978.428029] Lustre: server umount lustre-MDT0000 complete [ 2978.429654] LustreError: 167883:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2980.640847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.780393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.993305] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2984.740838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2984.856745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2986.071476] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 2987.778657] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.896301] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2989.518942] LustreError: 168324:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2989.522193] LustreError: 168324:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2989.525283] LustreError: 168324:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2990.048256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2990.048646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.049384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.055005] Lustre: Skipped 1 previous similar message [ 2995.169052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.172718] Lustre: Skipped 2 previous similar messages [ 2995.646410] LustreError: 169636:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2995.648196] LustreError: 169636:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2995.674065] Lustre: server umount lustre-MDT0000 complete [ 2996.886386] LustreError: 168277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237850 with bad export cookie 3003870167336355904 [ 2996.888782] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2996.891588] LustreError: 168277:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2996.954661] LustreError: 169842:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2996.956956] LustreError: 169842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2997.011990] Lustre: server umount lustre-MDT0001 complete [ 3000.413529] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3001.398464] Key type lgssc unregistered [ 3001.525533] LNet: 170373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.528536] LNetError: 170373:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3001.542369] LNet: Removed LNI 192.168.202.134@tcp [ 3001.816146] Key type .llcrypt unregistered [ 3001.817450] Key type ._llcrypt unregistered [ 3008.561341] Key type ._llcrypt registered [ 3008.562424] Key type .llcrypt registered [ 3008.600266] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3008.699183] Key type .llcrypt unregistered [ 3008.700341] Key type ._llcrypt unregistered [ 3015.010796] Key type ._llcrypt registered [ 3015.011702] Key type .llcrypt registered [ 3015.052417] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3015.374148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3015.415476] alg: No test for adler32 (adler32-zlib) [ 3016.283533] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3016.369024] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3017.951159] Key type lgssc registered [ 3018.308511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3020.915036] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3020.921196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3036.383318] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3041.503265] LustreError: 172440:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.134@tcp: failed processing log, type 4: rc = -110 [ 3068.127183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.093211] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3071.911193] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3074.213877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.412272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3074.434664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3075.784094] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3078.689206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3078.823511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3080.184301] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3082.017207] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3083.297836] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3084.240484] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3094.496471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3094.499066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.503624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.007985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.010210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.014064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3098.595068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3098.597394] Lustre: Skipped 1 previous similar message [ 3100.397083] LustreError: 175259:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3100.437586] Lustre: server umount lustre-OST0000 complete [ 3103.713904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.714737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.719623] Lustre: Skipped 1 previous similar message [ 3107.839617] LustreError: 175460:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3107.842382] LustreError: 175460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3107.869413] Lustre: server umount lustre-MDT0000 complete [ 3108.832116] LustreError: 173368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.837453] LustreError: 173368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3109.164229] LustreError: 172447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773237962 with bad export cookie 16618952245287035558 [ 3109.167305] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3109.169483] LustreError: 172447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3109.241097] LustreError: 175663:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3109.244157] LustreError: 175663:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3109.297671] Lustre: server umount lustre-MDT0001 complete [ 3112.934575] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3113.983452] Key type lgssc unregistered [ 3114.114411] LNet: 176195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3114.117444] LNetError: 176195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3114.128322] LNet: Removed LNI 192.168.202.134@tcp [ 3114.436263] Key type .llcrypt unregistered [ 3114.437341] Key type ._llcrypt unregistered [ 3123.616408] Key type ._llcrypt registered [ 3123.617613] Key type .llcrypt registered [ 3123.653936] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3123.998883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3124.058456] alg: No test for adler32 (adler32-zlib) [ 3124.911484] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3124.995625] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3126.575134] Key type lgssc registered [ 3126.933563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3129.592649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3129.599482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.772434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3132.087996] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3135.009252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3135.127537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3136.472404] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3138.253563] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3139.521714] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3141.917527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3142.068228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.074878] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3145.920849] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3156.395571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3157.938727] Lustre: Failing over lustre-MDT0000 [ 3158.013143] LustreError: 179880:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3158.039847] Lustre: server umount lustre-MDT0000 complete [ 3161.055969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3161.058119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.062325] LustreError: 178278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.069743] LustreError: 178278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3161.569353] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.570148] LustreError: 177560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.573675] Lustre: Skipped 1 previous similar message [ 3166.688194] LustreError: 178278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3166.692668] LustreError: 178278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3170.938970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.978719] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3171.068726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.090601] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3172.060695] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3172.347834] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3173.908603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3176.426490] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3176.434952] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3176.456629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3179.742217] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3191.264019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.267041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.272910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3191.780623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3193.132115] LustreError: 181209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3193.134461] LustreError: 181209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3193.172062] Lustre: server umount lustre-OST0000 complete [ 3196.897036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3196.897191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3196.900652] LustreError: Skipped 2 previous similar messages [ 3196.901193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3196.905128] Lustre: Skipped 2 previous similar messages [ 3196.910862] Lustre: Skipped 1 previous similar message [ 3200.641266] LustreError: 181414:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3200.644421] LustreError: 181414:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3200.671593] Lustre: server umount lustre-MDT0000 complete [ 3201.940474] LustreError: 179185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238055 with bad export cookie 13402542932041106746 [ 3201.943727] LustreError: 179185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3201.944108] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.068664] Lustre: server umount lustre-MDT0001 complete [ 3205.845545] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3206.956599] Key type lgssc unregistered [ 3207.093437] LNet: 182146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3207.096055] LNetError: 182146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.105398] LNet: Removed LNI 192.168.202.134@tcp [ 3207.435786] Key type .llcrypt unregistered [ 3207.436918] Key type ._llcrypt unregistered [ 3216.443193] Key type ._llcrypt registered [ 3216.445077] Key type .llcrypt registered [ 3216.489961] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3216.881288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3216.923033] alg: No test for adler32 (adler32-zlib) [ 3217.802195] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3217.894726] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3219.479169] Key type lgssc registered [ 3219.912867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3222.645437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3222.654320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.824338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.206958] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3228.172451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3228.301183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3229.650182] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3231.565172] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.797092] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3235.157743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.275648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.264287] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3239.106138] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3248.614876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3250.515762] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3258.068041] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3267.171699] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3279.328832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3279.329876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.331709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3279.337307] Lustre: Skipped 1 previous similar message [ 3284.449642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3284.451751] Lustre: Skipped 2 previous similar messages [ 3284.719072] LustreError: 185972:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3284.755374] Lustre: server umount lustre-OST0000 complete [ 3289.569329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.570611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.574493] Lustre: Skipped 1 previous similar message [ 3289.578158] Lustre: Skipped 1 previous similar message [ 3294.688125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3294.689785] Lustre: Skipped 1 previous similar message [ 3299.808691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.810372] Lustre: Skipped 1 previous similar message [ 3300.319165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3300.360324] LustreError: 186173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3300.362609] LustreError: 186173:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3300.389330] Lustre: server umount lustre-MDT0000 complete [ 3301.675962] LustreError: 183496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238155 with bad export cookie 6459595407470890907 [ 3301.678919] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3301.681075] LustreError: 183496:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3301.750263] LustreError: 186376:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3301.752415] LustreError: 186376:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3301.806778] Lustre: server umount lustre-MDT0001 complete [ 3305.553382] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3306.569936] Key type lgssc unregistered [ 3306.705530] LNet: 186907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.707995] LNetError: 186907:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.714264] LNet: Removed LNI 192.168.202.134@tcp [ 3307.032576] Key type .llcrypt unregistered [ 3307.033929] Key type ._llcrypt unregistered [ 3316.095687] Key type ._llcrypt registered [ 3316.096808] Key type .llcrypt registered [ 3316.133575] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3316.454345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3316.460509] alg: No test for adler32 (adler32-zlib) [ 3317.313119] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3317.395272] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3318.975117] Key type lgssc registered [ 3319.337786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3321.934219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3321.940779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.102189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.397417] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3327.214205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3327.355421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.648557] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3330.441736] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3331.734852] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3334.055505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3334.175969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.184919] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3337.975971] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3347.494572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3350.115441] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3361.604445] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3368.338240] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3375.989394] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3383.265375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3383.267491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3383.270657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3383.274122] Lustre: Skipped 1 previous similar message [ 3383.277364] Lustre: Skipped 1 previous similar message [ 3388.385895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.389145] Lustre: Skipped 1 previous similar message [ 3388.590091] LustreError: 192463:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3388.621042] Lustre: server umount lustre-OST0000 complete [ 3393.505754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.506326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.509868] Lustre: Skipped 1 previous similar message [ 3398.624555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.626459] Lustre: Skipped 2 previous similar messages [ 3403.744844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3403.747955] Lustre: Skipped 1 previous similar message [ 3404.255129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3404.297256] LustreError: 192665:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3404.300206] LustreError: 192665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3404.331601] Lustre: server umount lustre-MDT0000 complete [ 3405.536150] LustreError: 188260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238259 with bad export cookie 10748562680792081667 [ 3405.539490] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.539814] LustreError: 188260:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3405.610050] LustreError: 192867:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3405.611926] LustreError: 192867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3405.653944] Lustre: server umount lustre-MDT0001 complete [ 3409.333749] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3410.324714] Key type lgssc unregistered [ 3410.449377] LNet: 193407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3410.452187] LNetError: 193407:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3410.461349] LNet: Removed LNI 192.168.202.134@tcp [ 3410.735548] Key type .llcrypt unregistered [ 3410.736596] Key type ._llcrypt unregistered [ 3419.860567] Key type ._llcrypt registered [ 3419.861528] Key type .llcrypt registered [ 3419.898477] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3420.274789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3420.299733] alg: No test for adler32 (adler32-zlib) [ 3421.168265] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3421.266309] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3422.855190] Key type lgssc registered [ 3423.220881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3425.859893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3425.869076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.025504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.250041] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3430.999119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3431.129253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3432.470681] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3434.216099] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3435.415465] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3437.656168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3437.772911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.581829] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3441.394781] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3453.159569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3461.599885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3461.602460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3461.606896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3463.650340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3463.652180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3463.652531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.661837] Lustre: Skipped 1 previous similar message [ 3466.481135] LustreError: 197194:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3466.528439] Lustre: server umount lustre-OST0000 complete [ 3468.769671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3468.770442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.773546] Lustre: Skipped 1 previous similar message [ 3473.888858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3473.890806] Lustre: Skipped 2 previous similar messages [ 3479.008192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.010129] Lustre: Skipped 1 previous similar message [ 3482.079135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3482.114530] LustreError: 197397:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3482.116935] LustreError: 197397:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3482.143349] Lustre: server umount lustre-MDT0000 complete [ 3483.370406] LustreError: 196397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238337 with bad export cookie 15919412570160251727 [ 3483.374419] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3483.375465] LustreError: 196397:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3483.440392] LustreError: 197599:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3483.442390] LustreError: 197599:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3483.498904] Lustre: server umount lustre-MDT0001 complete [ 3487.307037] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3488.303345] Key type lgssc unregistered [ 3488.435441] LNet: 198130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3488.439850] LNetError: 198130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3488.448302] LNet: Removed LNI 192.168.202.134@tcp [ 3488.729863] Key type .llcrypt unregistered [ 3488.731173] Key type ._llcrypt unregistered [ 3495.792738] Key type ._llcrypt registered [ 3495.793759] Key type .llcrypt registered [ 3495.835755] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3496.337592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3496.376195] alg: No test for adler32 (adler32-zlib) [ 3497.241832] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3497.333776] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3498.919178] Key type lgssc registered [ 3499.310960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.914611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.920268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.082638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.396764] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3507.318494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3507.446207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.882645] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3510.838200] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3512.229847] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3514.637378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.789420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.886801] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3518.720309] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3528.102856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3533.284520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.285674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3533.290047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3533.294903] Lustre: Skipped 1 previous similar message [ 3534.895138] LustreError: 201770:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3534.932129] Lustre: server umount lustre-OST0000 complete [ 3538.300861] LustreError: 201973:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3538.303072] LustreError: 201973:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3538.329387] Lustre: server umount lustre-MDT0000 complete [ 3538.401068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3538.401416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.402271] LustreError: 199349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.402279] LustreError: 199349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3538.403821] LustreError: Skipped 1 previous similar message [ 3538.409175] Lustre: Skipped 2 previous similar messages [ 3539.603358] LustreError: 199334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238393 with bad export cookie 10517664640307416648 [ 3539.607067] LustreError: 199334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3539.607509] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3539.673907] LustreError: 202174:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3539.676462] LustreError: 202174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3539.729092] Lustre: server umount lustre-MDT0001 complete [ 3543.548875] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3544.544509] Key type lgssc unregistered [ 3544.676363] LNet: 202705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3544.678947] LNetError: 202705:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3544.687331] LNet: Removed LNI 192.168.202.134@tcp [ 3544.984801] Key type .llcrypt unregistered [ 3544.986093] Key type ._llcrypt unregistered [ 3552.199439] Key type ._llcrypt registered [ 3552.201314] Key type .llcrypt registered [ 3552.245036] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3552.579182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3552.589814] alg: No test for adler32 (adler32-zlib) [ 3553.445929] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3553.524676] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3555.103144] Key type lgssc registered [ 3555.466877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3558.049141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3558.055919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.217485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.533921] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3563.403859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.519584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.689212] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3566.298968] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3567.425482] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3569.519284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3569.640728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.318096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3571.571922] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3573.321783] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3576.772981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3576.803386] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3576.891220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3578.958635] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3580.821471] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3581.393724] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3592.030384] Lustre: Permanently deactivating lustre-OST0001 [ 3592.035850] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3600.376717] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3602.920524] Lustre: Permanently reactivating lustre-OST0001 [ 3602.927489] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3602.929450] Lustre: Skipped 2 previous similar messages [ 3608.057810] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.061996] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3608.065256] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3608.069284] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3608.081428] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3612.134776] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.135620] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3612.138384] Lustre: Skipped 2 previous similar messages [ 3615.148837] LustreError: 207365:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3615.167080] Lustre: server umount lustre-OST0001 complete [ 3617.252940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.253190] LustreError: 205525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3617.255059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.255068] Lustre: Skipped 1 previous similar message [ 3617.256769] Lustre: Skipped 1 previous similar message [ 3617.265449] LustreError: 205525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3622.369529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3622.370214] LustreError: 205523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.371587] Lustre: Skipped 2 previous similar messages [ 3622.376995] LustreError: 205523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3622.574058] LustreError: 207566:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3622.575987] LustreError: 207566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3622.611910] Lustre: server umount lustre-OST0000 complete [ 3624.927869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3624.930354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.934795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.608219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.611137] Lustre: Skipped 3 previous similar messages [ 3638.239145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3638.277035] LustreError: 207769:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3638.280284] LustreError: 207769:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3638.315108] Lustre: server umount lustre-MDT0000 complete [ 3639.604296] LustreError: 206431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238493 with bad export cookie 6196203872855311917 [ 3639.607613] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3639.610627] LustreError: 206431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3639.752532] Lustre: server umount lustre-MDT0001 complete [ 3643.623095] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3644.697636] Key type lgssc unregistered [ 3644.829364] LNet: 208503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3644.831926] LNetError: 208503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3644.845357] LNet: Removed LNI 192.168.202.134@tcp [ 3645.145310] Key type .llcrypt unregistered [ 3645.146452] Key type ._llcrypt unregistered [ 3656.103915] Key type ._llcrypt registered [ 3656.105536] Key type .llcrypt registered [ 3656.143078] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 3662.194800] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3662.569683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.600964] alg: No test for adler32 (adler32-zlib) [ 3663.461712] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3663.542950] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3665.127128] Key type lgssc registered [ 3665.501570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3668.349799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3670.401126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3672.437171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3674.549481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3676.869806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3676.894336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.900278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.005540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3678.018501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3678.055921] Lustre: lustre-MDT0000: new disk, initializing [ 3678.086903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.093523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3679.372588] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3683.462473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3683.487631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3683.521573] Lustre: 212422:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3683.549322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3683.551101] Lustre: Skipped 1 previous similar message [ 3683.588244] Lustre: lustre-MDT0001: new disk, initializing [ 3683.621773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3683.633594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3683.636721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3684.891150] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3688.061986] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3689.347712] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3691.654866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3691.676804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3691.769667] Lustre: lustre-OST0000: new disk, initializing [ 3691.772210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3691.803963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.643839] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3694.837403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3694.840866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3694.871916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.724495] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3698.095351] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3698.166290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3699.503762] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3699.593555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3702.317503] LustreError: 215359:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3702.353628] Lustre: server umount lustre-OST0000 complete [ 3704.288207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3704.291251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.295308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.312366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.313092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.433178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3710.435324] Lustre: Skipped 2 previous similar messages [ 3713.508282] LustreError: 215559:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3713.510610] LustreError: 215559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3713.540477] Lustre: server umount lustre-MDT0000 complete [ 3715.092629] LustreError: 212412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238568 with bad export cookie 2357618991458373842 [ 3715.096627] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.099189] LustreError: 212412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3725.283229] LustreError: 215762:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3725.285615] LustreError: 215762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3725.343582] Lustre: server umount lustre-MDT0001 complete [ 3728.354920] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:16:21 (1773238581) [ 3730.842870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.044392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.323945] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3735.197344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3735.321268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3736.566917] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3738.308050] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3739.531113] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3741.732793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3743.613678] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3745.367260] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3761.482559] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3771.098947] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3780.741833] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3786.240770] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3794.795607] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3811.908534] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3811.911080] Lustre: Skipped 1 previous similar message [ 3850.945691] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3850.948280] Lustre: Skipped 4 previous similar messages [ 3920.886792] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3920.888897] Lustre: Skipped 7 previous similar messages [ 3940.833166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3940.833885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.834525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3940.840662] Lustre: Skipped 1 previous similar message [ 3945.954656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.956524] Lustre: Skipped 2 previous similar messages [ 3946.031171] LustreError: 219990:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3946.034484] LustreError: 219990:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3946.070266] Lustre: server umount lustre-OST0000 complete [ 3950.559935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3950.563335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.343618] LustreError: 220196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3953.347917] LustreError: 220196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3953.376761] Lustre: server umount lustre-MDT0000 complete [ 3954.619294] LustreError: 216412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238808 with bad export cookie 2357618991458374710 [ 3954.622263] LustreError: 216412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3954.623132] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3954.738829] Lustre: server umount lustre-MDT0001 complete [ 3958.376885] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 3959.395281] Key type lgssc unregistered [ 3959.530434] LNet: 220929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3959.532985] LNetError: 220929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3959.544335] LNet: Removed LNI 192.168.202.134@tcp [ 3959.832918] Key type .llcrypt unregistered [ 3959.834021] Key type ._llcrypt unregistered [ 3968.293455] Key type ._llcrypt registered [ 3968.294482] Key type .llcrypt registered [ 3968.328461] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 3968.743863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3968.750493] alg: No test for adler32 (adler32-zlib) [ 3969.616742] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 3969.690954] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 3971.263180] Key type lgssc registered [ 3971.651750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3974.248328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3974.256421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.422730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.727550] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3979.531316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3979.674877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.953717] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3982.777712] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3984.088164] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3986.431775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3986.559322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.598509] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_default_debug -1 all [ 3990.509473] Lustre: DEBUG MARKER: oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4010.465349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.469426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.475938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.491221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4011.492238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4011.494208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.166749] LustreError: 224721:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4013.206901] Lustre: server umount lustre-OST0000 complete [ 4015.584188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4015.586496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.590225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.592071] Lustre: Skipped 1 previous similar message [ 4020.606180] LustreError: 224926:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4020.608953] LustreError: 224926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4020.639692] Lustre: server umount lustre-MDT0000 complete [ 4021.728092] LustreError: 222300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.732726] LustreError: 222300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4021.923253] LustreError: 222279:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1773238875 with bad export cookie 10496318673680886420 [ 4021.926176] LustreError: MGC192.168.202.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4021.928815] LustreError: 222279:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4021.998764] LustreError: 225128:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4022.000923] LustreError: 225128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4022.068961] Lustre: server umount lustre-MDT0001 complete [ 4025.668572] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4026.711886] Key type lgssc unregistered [ 4026.848458] LNet: 225660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4026.851246] LNetError: 225660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4026.860375] LNet: Removed LNI 192.168.202.134@tcp [ 4027.166729] Key type .llcrypt unregistered [ 4027.168322] Key type ._llcrypt unregistered [ 4034.255890] Key type ._llcrypt registered [ 4034.257284] Key type .llcrypt registered [ 4034.288486] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4034.383336] Key type .llcrypt unregistered [ 4034.384883] Key type ._llcrypt unregistered [ 4042.889491] Key type ._llcrypt registered [ 4042.890713] Key type .llcrypt registered [ 4042.935170] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing set_hostid [ 4049.516932] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing load_modules_local [ 4049.795832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4049.802982] alg: No test for adler32 (adler32-zlib) [ 4050.644307] Lustre: Lustre: Build Version: 2.16.59_50_g7c1f86b [ 4050.726085] LNet: Added LNI 192.168.202.134@tcp [8/256/0/180] [ 4052.303139] Key type lgssc registered [ 4052.603358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4055.519555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4057.764477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4059.963389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.341747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4062.987127] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3864 sec ========== 10:21:56 (1773238916) [ 4063.594766] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:21:57 (1773238917) === [ 4064.782235] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:21:58 (1773238918) === [ 4078.505524] Lustre: DEBUG MARKER: oleg234-server.virtnet: executing unload_modules_local [ 4079.491223] Key type lgssc unregistered [ 4079.614367] LNet: 231736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4079.616562] LNetError: 231736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4079.627284] LNet: Removed LNI 192.168.202.134@tcp [ 4079.910518] Key type .llcrypt unregistered [ 4079.911638] Key type ._llcrypt unregistered