[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516871022 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003253] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008899] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014000] *** VALIDATE selinux *** [ 0.019036] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024418] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025171] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027134] *** VALIDATE tmpfs *** [ 0.028600] *** VALIDATE proc *** [ 0.029334] *** VALIDATE cgroup *** [ 0.030013] *** VALIDATE cgroup2 *** [ 0.031267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034032] Spectre V2 : User space: Vulnerable [ 0.035000] Speculative Store Bypass: Vulnerable [ 0.037290] debug: unmapping init [mem 0xffffffff8fa59000-0xffffffff8fa60fff] [ 0.039183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041023] ... version: 2 [ 0.042014] ... bit width: 48 [ 0.043027] ... generic registers: 4 [ 0.044012] ... value mask: 0000ffffffffffff [ 0.045022] ... max period: 00007fffffffffff [ 0.046014] ... fixed-purpose events: 3 [ 0.046815] ... event mask: 000000070000000f [ 0.047364] rcu: Hierarchical SRCU implementation. [ 0.049623] smp: Bringing up secondary CPUs ... [ 0.050604] x86: Booting SMP configuration: [ 0.051040] .... node #0, CPUs: #1 #2 #3 [ 0.057754] smp: Brought up 1 node, 4 CPUs [ 0.059013] smpboot: Max logical packages: 1 [ 0.060014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138841] node 0 deferred pages initialised in 77ms [ 0.142173] devtmpfs: initialized [ 0.143253] x86/mm: Memory block size: 128MB [ 0.145717] gcov: version magic: 0x41383552 [ 0.149081] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152384] pinctrl core: initialized pinctrl subsystem [ 0.155225] [ 0.155866] ************************************************************* [ 0.158020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.162016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164015] ** ** [ 0.167017] ** This means that this kernel is built to expose internal ** [ 0.169018] ** IOMMU data structures, which may compromise security on ** [ 0.172021] ** your system. ** [ 0.174016] ** ** [ 0.176026] ** If you see this message and you are not debugging the ** [ 0.179018] ** kernel, report this immediately to your vendor! ** [ 0.181017] ** ** [ 0.183019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186014] ************************************************************* [ 0.188775] NET: Registered protocol family 16 [ 0.191602] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199414] cpuidle: using governor menu [ 0.201631] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204478] PCI: Using configuration type 1 for base access [ 0.206131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215137] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217155] cryptd: max_cpu_qlen set to 1000 [ 0.219062] ACPI: Added _OSI(Module Device) [ 0.220000] ACPI: Added _OSI(Processor Device) [ 0.223020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228588] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234531] ACPI: Interpreter enabled [ 0.236078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238016] ACPI: Using IOAPIC for interrupt routing [ 0.239138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243484] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254118] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266303] acpiphp: Slot [2] registered [ 0.268139] acpiphp: Slot [5] registered [ 0.269122] acpiphp: Slot [6] registered [ 0.270100] acpiphp: Slot [7] registered [ 0.272138] acpiphp: Slot [8] registered [ 0.273116] acpiphp: Slot [9] registered [ 0.274118] acpiphp: Slot [10] registered [ 0.276160] acpiphp: Slot [3] registered [ 0.276985] acpiphp: Slot [4] registered [ 0.277082] acpiphp: Slot [11] registered [ 0.279122] acpiphp: Slot [12] registered [ 0.280123] acpiphp: Slot [13] registered [ 0.282126] acpiphp: Slot [14] registered [ 0.283119] acpiphp: Slot [15] registered [ 0.284140] acpiphp: Slot [16] registered [ 0.286144] acpiphp: Slot [17] registered [ 0.288170] acpiphp: Slot [18] registered [ 0.289082] acpiphp: Slot [19] registered [ 0.290156] acpiphp: Slot [20] registered [ 0.292124] acpiphp: Slot [21] registered [ 0.293168] acpiphp: Slot [22] registered [ 0.295199] acpiphp: Slot [23] registered [ 0.296157] acpiphp: Slot [24] registered [ 0.297085] acpiphp: Slot [25] registered [ 0.299180] acpiphp: Slot [26] registered [ 0.300104] acpiphp: Slot [27] registered [ 0.302098] acpiphp: Slot [28] registered [ 0.303073] acpiphp: Slot [29] registered [ 0.304084] acpiphp: Slot [30] registered [ 0.306128] acpiphp: Slot [31] registered [ 0.308092] PCI host bridge to bus 0000:00 [ 0.309024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.314023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.323031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331250] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340802] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357535] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359730] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.367030] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.385018] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.390020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.396390] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403036] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422013] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.432827] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.439021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.459021] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.468000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.480016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.492023] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.500931] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.509021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.515026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.531022] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.543000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.549016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.555016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.568019] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.578816] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.585023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.590020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.602025] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.613892] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.615428] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.618399] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.620378] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.623309] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.628100] iommu: Default domain type: Passthrough [ 0.630681] SCSI subsystem initialized [ 0.632316] ACPI: bus type USB registered [ 0.633139] usbcore: registered new interface driver usbfs [ 0.635081] usbcore: registered new interface driver hub [ 0.637086] usbcore: registered new device driver usb [ 0.639206] pps_core: LinuxPPS API ver. 1 registered [ 0.641013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.644087] PTP clock support registered [ 0.646138] EDAC MC: Ver: 3.0.0 [ 0.647632] PCI: Using ACPI for IRQ routing [ 0.649985] NetLabel: Initializing [ 0.651018] NetLabel: domain hash size = 128 [ 0.653013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.655094] NetLabel: unlabeled traffic allowed by default [ 0.657153] vgaarb: loaded [ 0.658316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.661020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.666006] clocksource: Switched to clocksource kvm-clock [ 0.778552] VFS: Disk quotas dquot_6.6.0 [ 0.780275] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782815] *** VALIDATE ramfs *** [ 0.784298] *** VALIDATE hugetlbfs *** [ 0.786109] pnp: PnP ACPI init [ 0.788624] pnp: PnP ACPI: found 6 devices [ 0.812332] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.816181] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.818647] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.820741] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.823131] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.825956] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.828857] NET: Registered protocol family 2 [ 0.831588] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.836481] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.840511] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.846618] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.849896] TCP: Hash tables configured (established 65536 bind 65536) [ 0.852868] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.855729] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.858456] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.861484] NET: Registered protocol family 1 [ 0.865832] RPC: Registered named UNIX socket transport module. [ 0.867836] RPC: Registered udp transport module. [ 0.869127] RPC: Registered tcp transport module. [ 0.870881] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.873172] NET: Registered protocol family 44 [ 0.874731] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.877077] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.879377] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.881785] PCI: CLS 0 bytes, default 64 [ 0.883688] Unpacking initramfs... [ 2.373546] debug: unmapping init [mem 0xffff90a73cc54000-0xffff90a73ffbffff] [ 2.380445] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.383157] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.387586] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.927808] Initialise system trusted keyrings [ 2.929503] Key type blacklist registered [ 2.940221] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.950836] zbud: loaded [ 2.954245] *** VALIDATE nfs *** [ 2.955564] *** VALIDATE nfs4 *** [ 2.957428] pstore: using deflate compression [ 2.960682] Platform Keyring initialized [ 3.064664] NET: Registered protocol family 38 [ 3.066716] Key type asymmetric registered [ 3.068514] Asymmetric key parser 'x509' registered [ 3.070787] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.073560] io scheduler mq-deadline registered [ 3.075473] io scheduler kyber registered [ 3.078513] io scheduler bfq registered [ 3.080483] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.083447] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.086579] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.089438] ACPI: Power Button [PWRF] [ 3.185598] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.276795] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.458345] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.549239] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.732398] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.761769] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.793864] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.799432] Non-volatile memory driver v1.3 [ 3.801299] Linux agpgart interface v0.103 [ 3.834417] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.836779] vda: detected capacity change from 0 to 68624384 [ 3.850828] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.853842] vdb: detected capacity change from 0 to 1073741824 [ 3.869547] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.871972] vdc: detected capacity change from 0 to 2621440000 [ 3.886791] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.889591] vdd: detected capacity change from 0 to 2621440000 [ 3.905581] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.908996] vde: detected capacity change from 0 to 4294967296 [ 3.924827] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.928270] vdf: detected capacity change from 0 to 4294967296 [ 3.940330] libphy: Fixed MDIO Bus: probed [ 3.949950] usbcore: registered new interface driver usbserial_generic [ 3.951833] usbserial: USB Serial support registered for generic [ 3.953286] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.956391] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.958303] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.960489] mousedev: PS/2 mouse device common for all mice [ 3.963650] rtc_cmos 00:05: RTC can wake from S4 [ 3.967471] rtc_cmos 00:05: registered as rtc0 [ 3.968398] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.969371] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.974072] intel_pstate: CPU model not supported [ 3.978051] hid: raw HID events driver (C) Jiri Kosina [ 3.980343] usbcore: registered new interface driver usbhid [ 3.982789] usbhid: USB HID core driver [ 3.984573] drop_monitor: Initializing network drop monitor service [ 3.985060] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.987397] Initializing XFRM netlink socket [ 3.992344] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.992473] NET: Registered protocol family 10 [ 3.997980] Segment Routing with IPv6 [ 3.999751] NET: Registered protocol family 17 [ 4.001738] mpls_gso: MPLS GSO support [ 4.007898] RAS: Correctable Errors collector initialized. [ 4.010158] AVX version of gcm_enc/dec engaged. [ 4.011913] AES CTR mode by8 optimization enabled [ 4.083032] sched_clock: Marking stable (4083014058, 0)->(5046522813, -963508755) [ 4.086575] registered taskstats version 1 [ 4.088650] Loading compiled-in X.509 certificates [ 4.091848] zswap: loaded using pool lzo/zbud [ 4.119604] Key type big_key registered [ 4.131614] Key type encrypted registered [ 4.133792] ima: No TPM chip found, activating TPM-bypass! [ 4.136378] ima: Allocated hash algorithm: sha1 [ 4.138427] ima: No architecture policies found [ 4.140481] evm: Initialising EVM extended attributes: [ 4.142771] evm: security.selinux [ 4.144318] evm: security.ima [ 4.145554] evm: security.capability [ 4.147220] evm: HMAC attrs: 0x1 [ 4.150115] rtc_cmos 00:05: setting system clock to 2025-12-22 14:03:15 UTC (1766412195) [ 4.156856] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 4.160381] debug: unmapping init [mem 0xffffffff8f782000-0xffffffff8fa58fff] [ 4.169121] Write protecting the kernel read-only data: 28672k [ 4.172735] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 4.175793] debug: unmapping init [mem 0xffffffff8e714000-0xffffffff8e7fffff] [ 4.212756] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.221947] systemd[1]: Detected virtualization kvm. [ 4.223944] systemd[1]: Detected architecture x86-64. [ 4.226165] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.253644] systemd[1]: No hostname configured. [ 4.255629] systemd[1]: Set hostname to . [ 4.258042] random: systemd: uninitialized urandom read (16 bytes read) [ 4.260741] systemd[1]: Initializing machine ID from random generator. [ 4.312918] random: ln: uninitialized urandom read (6 bytes read) [ 4.418275] random: systemd: uninitialized urandom read (16 bytes read) [ 4.423438] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.432315] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.440447] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.151595] device-mapper: uevent: version 1.0.3 [ 5.153925] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.882412] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.948294] virtio_net virtio0 ens2: renamed from eth0 [ 5.963587] scsi host0: ata_piix [ 6.015746] scsi host1: ata_piix [ 6.017412] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.022750] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.564275] dracut-initqueue[597]: RTNETLINK answers: File exists [ 10.661044] random: crng init done [ 10.662731] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.274628] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.507474] printk: systemd: 21 output lines suppressed due to ratelimiting [ 12.789507] SELinux: Disabled at runtime. [ 12.851768] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.861789] systemd[1]: Detected virtualization kvm. [ 12.863784] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.399956] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.403704] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.407975] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.413306] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.420542] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.429339] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.434570] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 13.579745] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.891909] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.244266] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.285736] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.440668] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.452627] EDAC sbridge: Ver: 1.1.2 [ 16.272801] Key type dns_resolver registered [ 16.594652] NFS: Registering the id_resolver key type [ 16.596841] Key type id_resolver registered [ 16.598767] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg125-server login: [ 60.612762] libcfs: loading out-of-tree module taints kernel. [ 60.655228] Key type ._llcrypt registered [ 60.657073] Key type .llcrypt registered [ 60.784199] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 77.949915] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 79.099446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.117633] alg: No test for adler32 (adler32-zlib) [ 80.504159] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 81.157669] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 82.952197] Key type lgssc registered [ 84.594184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.897019] hrtimer: interrupt took 5007874 ns [ 100.910646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 102.607251] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.184985] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.612244] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.061809] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 139.187114] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 150.789271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.835418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 150.849350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.076284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 152.112165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 152.184373] Lustre: lustre-MDT0000: new disk, initializing [ 152.285582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.307862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.843080] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.654829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.748038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 168.859173] Lustre: 6522:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 168.885971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 168.888932] Lustre: Skipped 1 previous similar message [ 168.978526] Lustre: lustre-MDT0001: new disk, initializing [ 169.070086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.109424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.120901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.180146] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.201808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.377097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.417635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 185.617931] Lustre: lustre-OST0000: new disk, initializing [ 185.631485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.706486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.973332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.980053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.061568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.621845] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.848931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 201.921673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 202.025890] Lustre: lustre-OST0001: new disk, initializing [ 202.028672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 202.098350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.020171] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.410516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 208.420597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 208.488317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 217.052249] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.511355] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 236.713843] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.275418] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing yml_node [ 245.511898] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 247.825785] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 250.284702] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 251.840966] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Dec 22 09:07:22 EST 2025 [ 268.204881] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 269.792090] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.154946] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.161329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 285.172705] Lustre: Skipped 1 previous similar message [ 290.274152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.283450] Lustre: Skipped 6 previous similar messages [ 290.885975] LustreError: 12228:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 290.978932] Lustre: server umount lustre-MDT0000 complete [ 294.413696] LustreError: 6512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766412485 with bad export cookie 8865525124478759759 [ 294.417531] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.424956] LustreError: 6512:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 295.393553] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.396950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.411604] Lustre: Skipped 3 previous similar messages [ 299.426532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.429699] Lustre: Skipped 2 previous similar messages [ 300.663071] LustreError: 12429:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 300.670893] LustreError: 12429:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 300.852595] Lustre: server umount lustre-MDT0001 complete [ 304.426406] LustreError: 12630:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 304.432619] LustreError: 12630:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 304.495955] Lustre: server umount lustre-OST0000 complete [ 308.174837] LustreError: 12831:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 308.184290] LustreError: 12831:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 308.267662] Lustre: server umount lustre-OST0001 complete [ 314.290716] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 323.044077] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 332.233380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.713193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.634388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 348.761927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.192335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.246935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.433330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 354.497513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 354.606587] Lustre: lustre-MDT0000: new disk, initializing [ 354.729066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.759053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 358.626317] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 368.101104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 368.180794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 368.290132] Lustre: 15016:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 368.424519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 368.436104] Lustre: Skipped 1 previous similar message [ 368.561355] Lustre: lustre-MDT0001: new disk, initializing [ 368.704574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 368.744803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 368.749110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 372.155593] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 378.960859] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 382.102694] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 387.111280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 387.160308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 387.394765] Lustre: lustre-OST0000: new disk, initializing [ 387.398656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 387.474839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.244584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.249973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.381650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 392.289608] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 399.157661] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 402.498567] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 402.719733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 405.927536] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 406.092497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 410.084618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.091391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 410.098463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 410.111464] Lustre: Skipped 1 previous similar message [ 413.758239] LustreError: 17956:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 413.765750] LustreError: 17956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 413.889621] Lustre: server umount lustre-OST0000 complete [ 419.810335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 419.815598] LustreError: Skipped 1 previous similar message [ 419.819764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.828784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 419.831143] Lustre: Skipped 1 previous similar message [ 423.351098] LustreError: 18158:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 423.354172] LustreError: 18158:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 423.432522] Lustre: server umount lustre-MDT0000 complete [ 425.441771] LustreError: 15022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.454228] LustreError: 15022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 426.661039] LustreError: 15008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766412618 with bad export cookie 8865525124478761299 [ 426.671833] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 426.675394] LustreError: 15008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 427.097755] Lustre: server umount lustre-MDT0001 complete [ 430.354983] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:10:20 (1766412620) [ 435.040768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.330688] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.394083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.997128] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 440.804645] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.199197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 447.383537] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 451.632904] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 454.369948] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 459.546259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 464.000200] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 468.236414] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 479.210542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.211048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.216263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.222490] Lustre: Skipped 2 previous similar messages [ 479.236076] Lustre: Skipped 3 previous similar messages [ 483.524236] LustreError: 21128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 483.539361] LustreError: 21128:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 483.639391] Lustre: server umount lustre-OST0000 complete [ 489.441984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.451494] Lustre: Skipped 1 previous similar message [ 492.660114] Lustre: server umount lustre-MDT0000 complete [ 494.562345] LustreError: 18884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.576177] LustreError: 18884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 495.460389] LustreError: 18866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766412686 with bad export cookie 8865525124478762181 [ 495.463705] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.478716] LustreError: 18866:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 495.732832] Lustre: server umount lustre-MDT0001 complete [ 504.991569] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 507.403434] Key type lgssc unregistered [ 507.658685] LNet: 22066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.672423] LNetError: 22066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 507.720533] LNet: Removed LNI 192.168.201.125@tcp [ 508.524393] Key type .llcrypt unregistered [ 508.528445] Key type ._llcrypt unregistered [ 527.469635] Key type ._llcrypt registered [ 527.471170] Key type .llcrypt registered [ 527.564743] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 528.384255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 528.548608] alg: No test for adler32 (adler32-zlib) [ 529.600156] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 529.792362] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 531.448202] Key type lgssc registered [ 532.397215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 539.521601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 539.544771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.061589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.823977] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 551.395265] LustreError: 23438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.421805] LustreError: 23438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 553.376870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 553.888736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 557.513821] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 562.115992] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 564.827213] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.307237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.706255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.094511] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 579.793340] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 586.101383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 594.404878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.415495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.435499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 596.455340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.456403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 596.467888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 596.482988] Lustre: Skipped 1 previous similar message [ 598.272902] LustreError: 26128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 598.375533] Lustre: server umount lustre-OST0000 complete [ 601.571129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.572910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.582317] Lustre: Skipped 1 previous similar message [ 606.692221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 606.694946] Lustre: Skipped 2 previous similar messages [ 611.811318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.814023] Lustre: Skipped 1 previous similar message [ 615.392107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 615.461932] LustreError: 26329:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 615.467409] LustreError: 26329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 615.530528] Lustre: server umount lustre-MDT0000 complete [ 616.930178] LustreError: 23439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.941495] LustreError: 23439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 618.012638] LustreError: 24158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766412809 with bad export cookie 16630891947905748266 [ 618.022487] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.023688] LustreError: 24158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 618.137078] LustreError: 26532:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 618.145048] LustreError: 26532:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 618.259996] Lustre: server umount lustre-MDT0001 complete [ 625.470963] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 627.177953] Key type lgssc unregistered [ 627.406451] LNet: 27064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.410898] LNetError: 27064:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.427662] LNet: Removed LNI 192.168.201.125@tcp [ 628.051556] Key type .llcrypt unregistered [ 628.053426] Key type ._llcrypt unregistered [ 645.754540] Key type ._llcrypt registered [ 645.756928] Key type .llcrypt registered [ 645.838415] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 646.410079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.421630] alg: No test for adler32 (adler32-zlib) [ 647.412161] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 647.606785] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 649.248343] Key type lgssc registered [ 650.140659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 655.488230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 655.500169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.827518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.186967] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 665.320935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 665.671675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 668.726493] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 673.014953] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 676.390879] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 687.181605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 687.443694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 692.244272] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 696.962168] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 701.816770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 707.043498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.079518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 707.083078] Lustre: Skipped 1 previous similar message [ 708.607121] LustreError: 31030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 708.737565] Lustre: server umount lustre-OST0000 complete [ 716.769316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 716.775840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.784907] Lustre: Skipped 1 previous similar message [ 716.792226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 718.847826] LustreError: 31231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 718.853473] LustreError: 31231:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 718.962775] Lustre: server umount lustre-MDT0000 complete [ 722.182634] LustreError: 28419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766412913 with bad export cookie 17079172451460556740 [ 722.186715] LustreError: 28419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 722.186738] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 722.320291] LustreError: 31433:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 722.331551] LustreError: 31433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 722.457561] Lustre: server umount lustre-MDT0001 complete [ 730.459978] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 732.903692] Key type lgssc unregistered [ 733.189883] LNet: 31963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.196725] LNetError: 31963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.217636] LNet: Removed LNI 192.168.201.125@tcp [ 733.993643] Key type .llcrypt unregistered [ 733.996212] Key type ._llcrypt unregistered [ 758.169262] Key type ._llcrypt registered [ 758.171587] Key type .llcrypt registered [ 758.300504] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 759.178486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.229182] alg: No test for adler32 (adler32-zlib) [ 760.381561] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 760.585551] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 762.280246] Key type lgssc registered [ 763.151732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 770.155315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 770.172308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.848577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.555757] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 782.307525] LustreError: 33357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.329882] LustreError: 33357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 782.697976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 782.990618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 786.282611] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 791.670671] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 795.983317] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 801.847440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 802.165279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.982476] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 811.337921] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 818.554215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 831.978600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 832.000937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.045768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 834.025232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 834.026096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.029818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 837.500825] LustreError: 35622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 837.620187] Lustre: server umount lustre-OST0000 complete [ 844.258929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.259808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 844.261772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 844.269813] Lustre: Skipped 1 previous similar message [ 844.281868] Lustre: Skipped 2 previous similar messages [ 846.939638] LustreError: 35824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 846.942745] LustreError: 35824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 847.126113] Lustre: server umount lustre-MDT0000 complete [ 849.380416] LustreError: 33356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.908331] LustreError: 33336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766413041 with bad export cookie 5453623969207097183 [ 849.910167] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 849.926508] LustreError: 33336:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 850.063848] LustreError: 36025:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 850.068392] LustreError: 36025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 850.155639] Lustre: server umount lustre-MDT0001 complete [ 859.162945] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 861.793633] Key type lgssc unregistered [ 862.082139] LNet: 36558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.092942] LNetError: 36558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.104826] LNet: Removed LNI 192.168.201.125@tcp [ 862.776588] Key type .llcrypt unregistered [ 862.779395] Key type ._llcrypt unregistered [ 884.682796] Key type ._llcrypt registered [ 884.684340] Key type .llcrypt registered [ 884.806518] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 885.796261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.945018] alg: No test for adler32 (adler32-zlib) [ 887.064606] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 887.285749] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 888.944247] Key type lgssc registered [ 890.473722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 898.351370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 898.365413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.714664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.830541] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 910.306710] LustreError: 37969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.322246] LustreError: 37969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 910.618720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 911.185091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.295238] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 922.252874] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.713108] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 931.370113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 931.628896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.712568] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 940.381432] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 947.029037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 957.250108] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 957.426095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.438518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.471062] Lustre: Skipped 1 previous similar message [ 962.338311] Lustre: lustre-OST0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 962.341799] Lustre: Skipped 2 previous similar messages [ 963.218249] LustreError: 40219:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 963.386476] Lustre: server umount lustre-OST0000 complete [ 983.009957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 983.011454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.023880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.033871] Lustre: Skipped 1 previous similar message [ 983.061032] Lustre: Skipped 3 previous similar messages [ 987.499669] LustreError: 40598:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 987.517490] LustreError: 40598:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 987.651532] Lustre: server umount lustre-MDT0000 complete [ 988.129558] LustreError: 37951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.145304] LustreError: 37951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 989.157826] LustreError: 37946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.180314] LustreError: 37946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 991.418582] LustreError: 37930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766413182 with bad export cookie 16134509512909469470 [ 991.449161] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.669283] LustreError: 40800:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 997.687472] LustreError: 40800:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 997.862747] Lustre: server umount lustre-MDT0001 complete [ 1007.172507] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1009.515899] Key type lgssc unregistered [ 1009.777607] LNet: 41332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.796430] LNetError: 41332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.810577] LNet: Removed LNI 192.168.201.125@tcp [ 1010.597095] Key type .llcrypt unregistered [ 1010.599087] Key type ._llcrypt unregistered [ 1031.606801] Key type ._llcrypt registered [ 1031.608585] Key type .llcrypt registered [ 1031.693767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1032.329271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.371439] alg: No test for adler32 (adler32-zlib) [ 1033.403712] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1033.610345] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1035.280201] Key type lgssc registered [ 1036.279635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.822685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.848548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.257985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.162991] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1053.676491] LustreError: 42704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.697940] LustreError: 42704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1054.431651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1054.785077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1058.894426] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1064.646632] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1068.405411] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1075.193635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1075.633741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1081.115986] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1086.335233] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1088.943176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1098.041105] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.25@tcp (stopping) [ 1098.879142] LustreError: 44949:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1098.975954] Lustre: server umount lustre-MDT0000 complete [ 1099.236131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.242647] LustreError: 42699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.258032] Lustre: Skipped 2 previous similar messages [ 1099.285499] LustreError: 42699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1102.615311] LustreError: 42683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766413293 with bad export cookie 15502109542870369566 [ 1102.617805] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.626179] LustreError: 42683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1102.814282] LustreError: 45150:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1102.891304] LustreError: 45150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1103.067279] Lustre: server umount lustre-MDT0001 complete [ 1115.323504] LustreError: 42685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.25@tcp arrived at 1766413306 with bad export cookie 15502109542870370231 [ 1115.343115] LustreError: 42685:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1132.793767] LustreError: 45372:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1132.796207] LustreError: 45372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1132.961202] Lustre: server umount lustre-OST0000 complete [ 1147.977906] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1150.581339] Key type lgssc unregistered [ 1150.865976] LNet: 46211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.877149] LNetError: 46211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.920307] LNet: Removed LNI 192.168.201.125@tcp [ 1153.056805] Key type .llcrypt unregistered [ 1153.059450] Key type ._llcrypt unregistered [ 1178.585383] Key type ._llcrypt registered [ 1178.590452] Key type .llcrypt registered [ 1178.695907] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1180.114125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.210359] alg: No test for adler32 (adler32-zlib) [ 1181.261589] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1181.513722] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1183.248169] Key type lgssc registered [ 1184.411723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.428073] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.441123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1208.033265] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1213.152206] LustreError: 47578:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 1239.840191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.998690] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1251.351448] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1416.418861] LustreError: 48288:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1416.536692] Lustre: server umount lustre-OST0000 complete [ 1429.328323] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1431.782505] Key type lgssc unregistered [ 1432.060730] LNet: 49124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.071315] LNetError: 49124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.082476] LNet: Removed LNI 192.168.201.125@tcp [ 1432.738139] Key type .llcrypt unregistered [ 1432.743806] Key type ._llcrypt unregistered [ 1456.543309] Key type ._llcrypt registered [ 1456.544815] Key type .llcrypt registered [ 1456.644705] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1457.451936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.551067] alg: No test for adler32 (adler32-zlib) [ 1458.684569] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1458.921547] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1460.608247] Key type lgssc registered [ 1461.741136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.360317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.401552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.850203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.645672] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1481.193211] LustreError: 50520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.199685] LustreError: 50520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1481.747362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1482.038715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.029343] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1489.657515] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.069358] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1498.679513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1499.001434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.015265] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1507.345060] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1512.423480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1512.426893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.446777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.471195] LustreError: 52760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1515.704601] Lustre: server umount lustre-OST0000 complete [ 1521.635117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.643064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1521.661585] Lustre: Skipped 1 previous similar message [ 1525.845480] LustreError: 52962:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1525.849387] LustreError: 52962:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1525.914575] Lustre: server umount lustre-MDT0000 complete [ 1526.754205] LustreError: 51589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.772203] LustreError: 51589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1529.075903] LustreError: 50500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766413720 with bad export cookie 568991032166024094 [ 1529.083707] LustreError: 50500:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1529.084188] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1529.201378] LustreError: 53164:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1529.204535] LustreError: 53164:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1529.343758] Lustre: server umount lustre-MDT0001 complete [ 1537.407880] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1540.339960] Key type lgssc unregistered [ 1540.593149] LNet: 53697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.603075] LNetError: 53697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.625493] LNet: Removed LNI 192.168.201.125@tcp [ 1541.381661] Key type .llcrypt unregistered [ 1541.386206] Key type ._llcrypt unregistered [ 1561.942058] Key type ._llcrypt registered [ 1561.943494] Key type .llcrypt registered [ 1562.081539] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1562.847175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.856892] alg: No test for adler32 (adler32-zlib) [ 1563.967588] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1564.230765] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1565.872853] Key type lgssc registered [ 1566.906516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1574.838359] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1574.852650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1590.432320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1595.552250] LustreError: 55062:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 1622.241059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1627.003385] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1631.515574] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1637.738392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.210575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.297391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1641.684746] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1643.496017] LustreError: 55088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1643.517454] LustreError: 55088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1648.636749] LustreError: 55088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1649.499550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1649.919521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1653.533862] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1658.633235] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1661.983638] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1663.980923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.997307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.007913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1665.510766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1665.520608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.532214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1667.556654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1672.673990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1672.677638] Lustre: Skipped 2 previous similar messages [ 1677.794561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1677.802334] Lustre: Skipped 1 previous similar message [ 1678.307254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1678.372143] LustreError: 57337:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1678.495610] Lustre: server umount lustre-OST0000 complete [ 1698.275057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.294201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1698.298551] Lustre: Skipped 1 previous similar message [ 1703.504204] LustreError: 57719:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1703.513354] LustreError: 57719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1703.606984] Lustre: server umount lustre-MDT0000 complete [ 1706.984522] LustreError: 57467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766413898 with bad export cookie 7400258826068442191 [ 1706.988343] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1707.002131] LustreError: 57467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1713.258282] LustreError: 57920:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1713.274686] LustreError: 57920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1713.485923] Lustre: server umount lustre-MDT0001 complete [ 1724.506209] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1726.859835] Key type lgssc unregistered [ 1727.162760] LNet: 58452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.172797] LNetError: 58452:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.192847] LNet: Removed LNI 192.168.201.125@tcp [ 1727.780640] Key type .llcrypt unregistered [ 1727.782860] Key type ._llcrypt unregistered [ 1747.499321] Key type ._llcrypt registered [ 1747.502571] Key type .llcrypt registered [ 1747.618773] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1748.625713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1748.676974] alg: No test for adler32 (adler32-zlib) [ 1749.789401] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1749.978696] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1751.650092] Key type lgssc registered [ 1752.950701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1760.169433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1760.195120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.641846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.266329] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1770.982395] LustreError: 59827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1770.989681] LustreError: 59827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1772.347915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1772.686328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1775.922685] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1780.710760] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1783.834354] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1789.378812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1789.759777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.156780] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1800.962773] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1806.138765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1816.546917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1816.573614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.665861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1821.671693] Lustre: Skipped 2 previous similar messages [ 1822.267100] LustreError: 62092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1822.466677] Lustre: server umount lustre-OST0000 complete [ 1826.790126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.796057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1826.797844] Lustre: Skipped 2 previous similar messages [ 1831.906330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.930820] Lustre: Skipped 2 previous similar messages [ 1832.178724] LustreError: 62295:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1832.203635] LustreError: 62295:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1832.371919] Lustre: server umount lustre-MDT0000 complete [ 1835.730887] LustreError: 59806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414027 with bad export cookie 9196318432438043505 [ 1835.735667] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.748962] LustreError: 59806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1835.896306] LustreError: 62497:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1835.910325] LustreError: 62497:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1836.147539] Lustre: server umount lustre-MDT0001 complete [ 1845.502318] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 1848.396844] Key type lgssc unregistered [ 1848.660557] LNet: 63031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.672110] LNetError: 63031:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.693583] LNet: Removed LNI 192.168.201.125@tcp [ 1849.570536] Key type .llcrypt unregistered [ 1849.572207] Key type ._llcrypt unregistered [ 1871.823376] Key type ._llcrypt registered [ 1871.831872] Key type .llcrypt registered [ 1871.947664] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 1872.631395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.660650] alg: No test for adler32 (adler32-zlib) [ 1873.665798] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1873.835660] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 1875.496201] Key type lgssc registered [ 1876.458555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.927444] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:34:35 (1766414075) [ 1890.574405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1890.598914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.912858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.207301] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1901.917319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1902.049139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1902.176716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.181574] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1910.173591] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1914.210278] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1921.158777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1921.448368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.608297] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1932.995908] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1936.826779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1943.415301] Lustre: Failing over lustre-MDT0000 [ 1943.520219] LustreError: 66941:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1943.696186] Lustre: server umount lustre-MDT0000 complete [ 1944.903936] LustreError: 64693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.914446] LustreError: 64693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1947.105074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.111895] LustreError: 64691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.126057] Lustre: Skipped 2 previous similar messages [ 1947.140067] LustreError: 64691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1949.992159] LustreError: 64691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.007072] LustreError: 64691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1951.819490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.967325] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.061570] Lustre: *** cfs_fail_loc=135, val=0*** [ 1952.141041] LustreError: 67451:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1952.149275] LustreError: 67451:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -2 [ 1952.161836] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1952.161836] [ 1952.168695] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1952.178818] LustreError: 67439:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1952.184545] LustreError: 67439:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1952.188805] LustreError: 67439:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1952.193215] LustreError: 67439:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 1952.218625] Lustre: server umount lustre-MDT0000 complete [ 1952.220907] LustreError: 67439:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1952.227344] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.242487] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1957.351368] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.358168] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1958.109264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.309104] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.575895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.620634] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1960.229451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1962.260639] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 1964.021977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1964.057047] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1964.122222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1969.638070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1969.658134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.681484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.729164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.748657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.780135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.607693] LustreError: 68262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1975.618899] LustreError: 68262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1975.764818] Lustre: server umount lustre-OST0000 complete [ 1979.361925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.372870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.373732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1979.399744] Lustre: Skipped 1 previous similar message [ 1979.434018] Lustre: Skipped 3 previous similar messages [ 1984.494200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.500676] Lustre: Skipped 1 previous similar message [ 1985.276750] LustreError: 68463:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1985.279201] LustreError: 68463:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1985.400240] Lustre: server umount lustre-MDT0000 complete [ 1988.655261] LustreError: 68464:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414180 with bad export cookie 4207221793194196375 [ 1988.661846] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.789587] LustreError: 68665:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1988.792437] LustreError: 68665:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1989.024777] Lustre: server umount lustre-MDT0001 complete [ 1997.546494] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2000.159888] Key type lgssc unregistered [ 2000.417250] LNet: 69196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.425889] LNetError: 69196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.445733] LNet: Removed LNI 192.168.201.125@tcp [ 2001.297863] Key type .llcrypt unregistered [ 2001.300564] Key type ._llcrypt unregistered [ 2022.670125] Key type ._llcrypt registered [ 2022.671321] Key type .llcrypt registered [ 2022.768931] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2023.915386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2023.959949] alg: No test for adler32 (adler32-zlib) [ 2024.973817] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2025.199524] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2026.913965] Key type lgssc registered [ 2028.073235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2035.473506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2035.502200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.887332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.497141] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2047.457524] LustreError: 70581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.470958] LustreError: 70581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2047.950207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2048.190432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.438623] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2056.667550] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.679266] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2064.647700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2064.983164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.955408] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2073.309693] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2082.426631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2089.343155] Lustre: Failing over lustre-MDT0000 [ 2089.443393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2089.453609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.460990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.467404] LustreError: 72852:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2089.583382] Lustre: server umount lustre-MDT0000 complete [ 2090.297323] LustreError: 70576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.026540] LustreError: 70581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.027514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.038088] LustreError: 70581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2093.047817] Lustre: Skipped 1 previous similar message [ 2095.391056] LustreError: 70578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.407039] LustreError: 70578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2096.371498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.456980] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.564982] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2096.673723] LustreError: 73361:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2096.680721] LustreError: 73361:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -9 [ 2096.696937] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2096.696937] [ 2096.710295] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2096.731234] LustreError: 73349:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2096.743028] LustreError: 73349:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2096.753394] LustreError: 73349:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2096.764155] LustreError: 73349:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 2096.789223] Lustre: server umount lustre-MDT0000 complete [ 2096.791478] LustreError: 73349:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2100.515344] LustreError: 70576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.551607] LustreError: 70576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2102.612913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.794691] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.801874] LustreError: 70574:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2103.111550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.174524] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2105.630964] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2106.707492] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2108.395481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2108.405137] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2108.446302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2113.513647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.515207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.517064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.522322] Lustre: Skipped 1 previous similar message [ 2114.933406] LustreError: 74173:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2114.939983] LustreError: 74173:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2115.070696] Lustre: server umount lustre-OST0000 complete [ 2118.625651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2118.631162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.639014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.650297] Lustre: Skipped 1 previous similar message [ 2118.652209] Lustre: Skipped 1 previous similar message [ 2123.745418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.761504] Lustre: Skipped 2 previous similar messages [ 2124.533460] LustreError: 74373:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2124.540035] LustreError: 74373:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2124.607714] Lustre: server umount lustre-MDT0000 complete [ 2127.772100] LustreError: 70563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414319 with bad export cookie 10829331112319107447 [ 2127.776527] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.776949] LustreError: 70563:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2127.940217] LustreError: 74575:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2127.943818] LustreError: 74575:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2128.091262] Lustre: server umount lustre-MDT0001 complete [ 2137.434636] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2139.705053] Key type lgssc unregistered [ 2140.001699] LNet: 75107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.013084] LNetError: 75107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.030993] LNet: Removed LNI 192.168.201.125@tcp [ 2140.685456] Key type .llcrypt unregistered [ 2140.688947] Key type ._llcrypt unregistered [ 2155.324138] Key type ._llcrypt registered [ 2155.327119] Key type .llcrypt registered [ 2155.476821] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2155.862502] Key type .llcrypt unregistered [ 2155.864598] Key type ._llcrypt unregistered [ 2178.809306] Key type ._llcrypt registered [ 2178.811583] Key type .llcrypt registered [ 2178.924637] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2179.777537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.841557] alg: No test for adler32 (adler32-zlib) [ 2180.888409] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2181.124755] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2182.768147] Key type lgssc registered [ 2183.556132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2191.076161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2191.132382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.801802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.077944] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2203.108992] LustreError: 76985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.147757] LustreError: 76985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2205.208830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2205.622092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.345928] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2215.082829] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2219.296624] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2226.785330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2227.297259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.240468] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2238.571893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2239.766733] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.191227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.191261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2249.217070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2249.220671] Lustre: Skipped 1 previous similar message [ 2254.028209] LustreError: 79232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2254.226531] Lustre: server umount lustre-OST0000 complete [ 2259.428946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.437398] Lustre: Skipped 1 previous similar message [ 2259.441690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2259.454797] Lustre: Skipped 1 previous similar message [ 2263.669623] LustreError: 79433:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2263.679646] LustreError: 79433:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2263.771241] Lustre: server umount lustre-MDT0000 complete [ 2264.547677] LustreError: 78002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.562281] LustreError: 78002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2267.139641] LustreError: 76966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414458 with bad export cookie 3213795173570408019 [ 2267.148197] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.151678] LustreError: 76966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2267.267879] LustreError: 79635:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2267.272770] LustreError: 79635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2267.481825] Lustre: server umount lustre-MDT0001 complete [ 2275.244446] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2277.247173] Key type lgssc unregistered [ 2277.493637] LNet: 80167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.502845] LNetError: 80167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.518802] LNet: Removed LNI 192.168.201.125@tcp [ 2278.213304] Key type .llcrypt unregistered [ 2278.214793] Key type ._llcrypt unregistered [ 2300.785658] Key type ._llcrypt registered [ 2300.787213] Key type .llcrypt registered [ 2300.987509] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2301.853061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.892431] alg: No test for adler32 (adler32-zlib) [ 2303.051943] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2303.295315] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2305.064312] Key type lgssc registered [ 2306.398628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2313.755096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2313.765677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.258558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.118218] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2325.474245] LustreError: 81562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.500723] LustreError: 81562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2326.594317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2326.978893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2330.470112] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2335.853597] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2339.749679] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2346.371521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2346.670707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.374811] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2356.955263] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2359.998102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2367.461152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.464768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.478851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.530129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.536341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.539722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.525276] LustreError: 83807:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2373.667553] Lustre: server umount lustre-OST0000 complete [ 2378.208939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.212387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.235891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.242987] Lustre: Skipped 1 previous similar message [ 2380.769152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2380.776657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.748230] LustreError: 84010:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2382.750983] LustreError: 84010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2382.871649] Lustre: server umount lustre-MDT0000 complete [ 2385.896583] LustreError: 81563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.921108] LustreError: 81563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2386.712181] LustreError: 81542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414578 with bad export cookie 7613802991643602653 [ 2386.719955] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.738323] LustreError: 81542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2386.857721] LustreError: 84213:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2386.863543] LustreError: 84213:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2387.061704] Lustre: server umount lustre-MDT0001 complete [ 2395.639303] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2397.776504] Key type lgssc unregistered [ 2398.046810] LNet: 84744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.065583] LNetError: 84744:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.155834] LNet: Removed LNI 192.168.201.125@tcp [ 2399.759520] Key type .llcrypt unregistered [ 2399.762096] Key type ._llcrypt unregistered [ 2419.407502] Key type ._llcrypt registered [ 2419.409148] Key type .llcrypt registered [ 2419.528709] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2420.493710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.576734] alg: No test for adler32 (adler32-zlib) [ 2421.568763] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2421.738964] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2423.392953] Key type lgssc registered [ 2424.348963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2431.581408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2431.622945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.032777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2437.070591] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2443.235925] LustreError: 86117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.262268] LustreError: 86117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2446.080627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2446.609826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2451.532953] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2457.610433] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2461.802359] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2468.110540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2468.371724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.597955] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2478.687868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2479.253593] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2491.874313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2491.891944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.933878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.919895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2492.937988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.986940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.290074] LustreError: 88364:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2497.451864] Lustre: server umount lustre-OST0000 complete [ 2503.141171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2503.160996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.198053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.206319] Lustre: Skipped 2 previous similar messages [ 2507.053244] LustreError: 88566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2507.056227] LustreError: 88566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2507.110741] Lustre: server umount lustre-MDT0000 complete [ 2509.289291] LustreError: 88276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.311302] LustreError: 88276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2510.571251] LustreError: 86096:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414701 with bad export cookie 5352043951678821061 [ 2510.588375] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2510.709783] LustreError: 88768:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2510.738608] LustreError: 88768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2510.907170] Lustre: server umount lustre-MDT0001 complete [ 2520.140693] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2522.690530] Key type lgssc unregistered [ 2522.913261] LNet: 89300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.920373] LNetError: 89300:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.935745] LNet: Removed LNI 192.168.201.125@tcp [ 2523.594571] Key type .llcrypt unregistered [ 2523.596383] Key type ._llcrypt unregistered [ 2546.201534] Key type ._llcrypt registered [ 2546.203831] Key type .llcrypt registered [ 2546.281865] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2547.299745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2547.333761] alg: No test for adler32 (adler32-zlib) [ 2548.351811] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2548.578042] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2550.280181] Key type lgssc registered [ 2551.245344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.452337] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.466447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2574.048393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2579.168224] LustreError: 90664:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 2605.856225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.604181] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2614.929506] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2618.921897] LustreError: 91521:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2618.999189] Lustre: server umount lustre-OST0000 complete [ 2626.541831] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 09:46:57 (1766414817) [ 2631.344746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.628144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.503802] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2636.769701] LustreError: 92194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.785107] LustreError: 92194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2640.866526] LustreError: 92194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.767335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2642.171775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.711619] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2650.434542] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2653.776364] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2658.702876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2658.935428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.005038] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2669.365821] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2676.366802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2687.969182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2687.987889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.012930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.069637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.072164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2692.075556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.497095] LustreError: 94619:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2693.508205] LustreError: 94619:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2693.586520] Lustre: server umount lustre-OST0000 complete [ 2697.187807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.191080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2697.200432] Lustre: Skipped 1 previous similar message [ 2697.203159] Lustre: Skipped 1 previous similar message [ 2702.238308] LustreError: 94820:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2702.246770] LustreError: 94820:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2702.309336] LustreError: 92193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.344661] LustreError: 92193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2702.418776] Lustre: server umount lustre-MDT0000 complete [ 2705.569952] LustreError: 94821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766414896 with bad export cookie 4153412659205977988 [ 2705.576931] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.749497] LustreError: 95023:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2705.753283] LustreError: 95023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2705.920599] Lustre: server umount lustre-MDT0001 complete [ 2715.114148] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2717.534219] Key type lgssc unregistered [ 2717.808605] LNet: 95556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.826966] LNetError: 95556:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.872352] LNet: Removed LNI 192.168.201.125@tcp [ 2718.595234] Key type .llcrypt unregistered [ 2718.597762] Key type ._llcrypt unregistered [ 2738.686588] Key type ._llcrypt registered [ 2738.688608] Key type .llcrypt registered [ 2738.774320] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2739.707466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2739.727442] alg: No test for adler32 (adler32-zlib) [ 2740.752046] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2740.924054] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2742.608328] Key type lgssc registered [ 2743.644755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2750.588462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2750.602559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.019946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.769787] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2762.209164] LustreError: 96930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.230590] LustreError: 96930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2763.815439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2764.153482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2767.657199] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2773.000116] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2776.906233] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.579460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.877967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.446910] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2792.006968] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2797.233817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2804.711622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2804.725925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.734542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.058246] LustreError: 99177:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2805.204257] Lustre: server umount lustre-OST0000 complete [ 2810.344372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.354211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.372615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.903941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.911899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.019789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.022750] Lustre: Skipped 1 previous similar message [ 2819.055562] LustreError: 99379:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2819.063686] LustreError: 99379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2819.128421] Lustre: server umount lustre-MDT0000 complete [ 2822.561501] LustreError: 99178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415013 with bad export cookie 14433673010393012889 [ 2822.566868] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2822.593599] LustreError: 99178:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2822.826565] LustreError: 99583:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2822.833111] LustreError: 99583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2823.006931] Lustre: server umount lustre-MDT0001 complete [ 2833.744875] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 2835.924991] Key type lgssc unregistered [ 2836.196096] LNet: 100115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2836.205148] LNetError: 100115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2836.223367] LNet: Removed LNI 192.168.201.125@tcp [ 2837.000631] Key type .llcrypt unregistered [ 2837.003938] Key type ._llcrypt unregistered [ 2851.858358] Key type ._llcrypt registered [ 2851.859896] Key type .llcrypt registered [ 2851.981621] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2852.981181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2852.994288] alg: No test for adler32 (adler32-zlib) [ 2853.991279] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2854.208355] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 2855.907300] Key type lgssc registered [ 2856.865370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2863.080431] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2863.093592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2878.624370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2883.744255] LustreError: 101314:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 2910.496647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.713746] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 2920.388420] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2926.015902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.222126] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2926.240824] LustreError: 102228:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2926.246335] LustreError: 102228:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2926.253357] LustreError: 102228:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2926.328341] Lustre: server umount lustre-MDT0000 complete [ 2926.330728] LustreError: 102228:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2947.555167] Lustre: 100656:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1766415122/real 1766415122] req@00000000a0d73255 x1852220422229120/t0(0) o400->MGC192.168.201.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1766415138 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2947.585916] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2947.600345] LustreError: 101327:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2968.034192] LustreError: 102658:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2968.156102] Lustre: server umount lustre-OST0000 complete [ 2975.964058] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 2983.257609] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 2991.141714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2997.133712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3003.210184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.495301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.166965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.245889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.460966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3014.496619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3014.573157] Lustre: lustre-MDT0000: new disk, initializing [ 3014.650418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.666049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3017.813261] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3026.292849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.365720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3026.447143] Lustre: 105004:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3026.501966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3026.504807] Lustre: Skipped 1 previous similar message [ 3026.561042] Lustre: lustre-MDT0001: new disk, initializing [ 3026.636460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3026.663127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3026.674287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3029.802381] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3037.099176] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.596123] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3046.329290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3046.390248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3046.608402] Lustre: lustre-OST0000: new disk, initializing [ 3046.611368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3046.699149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.687695] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3055.165435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3055.176836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3055.300111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3057.429292] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3060.282962] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3060.466615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3063.806816] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3064.010706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3065.826593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.831869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3065.839094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.850070] Lustre: Skipped 1 previous similar message [ 3070.948159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3070.962864] Lustre: Skipped 1 previous similar message [ 3071.738165] LustreError: 107947:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3071.740581] LustreError: 107947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3071.827782] Lustre: server umount lustre-OST0000 complete [ 3076.065442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.081184] Lustre: Skipped 1 previous similar message [ 3076.090606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.187785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.190732] Lustre: Skipped 1 previous similar message [ 3081.517663] LustreError: 108149:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3081.528969] LustreError: 108149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3081.609340] Lustre: server umount lustre-MDT0000 complete [ 3084.540079] LustreError: 104995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415275 with bad export cookie 14915820769524976784 [ 3084.552641] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.556966] LustreError: 104995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3084.681908] LustreError: 108350:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3084.687052] LustreError: 108350:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3084.846826] Lustre: server umount lustre-MDT0001 complete [ 3092.700710] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 09:54:43 (1766415283) [ 3095.937459] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3110.903224] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3119.385767] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3128.105432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.288049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3138.267036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.428487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3150.888430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3150.962938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.309804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3151.378827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3151.456918] Lustre: lustre-MDT0000: new disk, initializing [ 3151.553420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.585373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3154.894919] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3163.767526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.854097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3163.967533] Lustre: 111484:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3164.048161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3164.059430] Lustre: Skipped 1 previous similar message [ 3164.173675] Lustre: lustre-MDT0001: new disk, initializing [ 3164.282713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.336682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3164.348931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3167.626804] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3174.025518] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3177.431529] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3183.559903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3183.637780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3183.947867] Lustre: lustre-OST0000: new disk, initializing [ 3183.961893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3184.045425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.347657] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3192.385979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3192.397778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3192.513298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3195.864665] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3199.275068] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3199.534690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3203.187249] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3203.401931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3208.161900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3208.165782] LustreError: Skipped 1 previous similar message [ 3208.170626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.191324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.202156] Lustre: Skipped 2 previous similar messages [ 3211.342121] LustreError: 114422:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3211.357680] LustreError: 114422:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3211.509669] Lustre: server umount lustre-OST0000 complete [ 3215.332944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3215.347728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.407266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.409957] Lustre: Skipped 3 previous similar messages [ 3221.165120] LustreError: 114623:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3221.168402] LustreError: 114623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3221.234268] Lustre: server umount lustre-MDT0000 complete [ 3224.130114] LustreError: 114624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415415 with bad export cookie 14915820769524977666 [ 3224.132327] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.144867] LustreError: 114624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3224.405175] Lustre: server umount lustre-MDT0001 complete [ 3231.658507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.065302] LustreError: 115303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3232.082875] LustreError: 115303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3232.172845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.855858] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3237.348821] LustreError: 115303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.469188] LustreError: 115303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.900322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3246.103255] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3250.322266] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3253.184662] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3258.695971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3263.830358] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3268.343033] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3284.304432] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3288.546099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3288.550879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.558731] Lustre: Skipped 1 previous similar message [ 3288.565306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.691423] LustreError: 117666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3293.701719] LustreError: 117666:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3293.807190] Lustre: server umount lustre-OST0000 complete [ 3297.765494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.786214] Lustre: Skipped 2 previous similar messages [ 3302.868147] Lustre: server umount lustre-MDT0000 complete [ 3302.885437] LustreError: 117427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.903796] LustreError: 117427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3305.696351] LustreError: 115284:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415497 with bad export cookie 14915820769524978548 [ 3305.697332] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.715275] LustreError: 115284:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3305.942175] Lustre: server umount lustre-MDT0001 complete [ 3314.505265] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 3317.011659] Key type lgssc unregistered [ 3317.259766] LNet: 118602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3317.269256] LNetError: 118602:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3317.282924] LNet: Removed LNI 192.168.201.125@tcp [ 3318.076407] Key type .llcrypt unregistered [ 3318.081157] Key type ._llcrypt unregistered [ 3333.931149] Key type ._llcrypt registered [ 3333.932802] Key type .llcrypt registered [ 3334.045557] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 3344.596493] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 3345.221367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3345.244870] alg: No test for adler32 (adler32-zlib) [ 3346.197380] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 3346.362260] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 3347.984202] Key type lgssc registered [ 3348.781749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3355.211298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3360.223802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3365.165290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3370.058279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3375.736055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3375.776245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3375.788076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.989709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3377.020381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3377.068839] Lustre: lustre-MDT0000: new disk, initializing [ 3377.160282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.177539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3380.337978] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3389.242897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.294928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3389.373182] Lustre: 121708:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3389.417246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3389.421755] Lustre: Skipped 1 previous similar message [ 3389.466771] Lustre: lustre-MDT0001: new disk, initializing [ 3389.537105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.571369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3389.578607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3392.458488] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3398.670979] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3401.835918] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.789174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3407.866556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3408.170948] Lustre: lustre-OST0000: new disk, initializing [ 3408.184623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3408.304310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.645306] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3418.182093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3418.191099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3418.292186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3419.888173] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3422.987149] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3423.186996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3426.394405] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3426.570518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3428.834212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.838726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.854449] Lustre: Skipped 1 previous similar message [ 3428.873034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3433.966556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3433.969824] Lustre: Skipped 1 previous similar message [ 3434.054084] LustreError: 124645:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3434.201773] Lustre: server umount lustre-OST0000 complete [ 3439.073470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.081442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.087064] Lustre: Skipped 1 previous similar message [ 3443.641476] LustreError: 124846:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3443.655881] LustreError: 124846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3443.752215] Lustre: server umount lustre-MDT0000 complete [ 3446.775695] LustreError: 121699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415638 with bad export cookie 3658737596542623476 [ 3446.781479] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.913883] LustreError: 125048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3446.922430] LustreError: 125048:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3447.065756] Lustre: server umount lustre-MDT0001 complete [ 3453.866979] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:00:44 (1766415644) [ 3459.806671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3460.310529] LustreError: 125719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.319660] LustreError: 125719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3460.400273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.967307] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3465.699156] LustreError: 125719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.796900] LustreError: 125719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.408511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3471.696443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3475.141273] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3479.211905] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3481.911938] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.179893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.182638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3486.191809] Lustre: Skipped 2 previous similar messages [ 3486.202344] Lustre: Skipped 1 previous similar message [ 3489.740314] LustreError: 127074:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3489.743555] LustreError: 127074:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3489.802734] Lustre: server umount lustre-MDT0000 complete [ 3491.297898] LustreError: 126443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.309961] LustreError: 126443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3492.710551] LustreError: 125700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415684 with bad export cookie 3658737596542624358 [ 3492.716419] LustreError: 125700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3492.719578] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3492.927827] Lustre: server umount lustre-MDT0001 complete [ 3499.301582] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:01:29 (1766415689) [ 3504.710885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3520.224707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3525.345257] LustreError: 127922:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 3551.008368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.700619] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3560.205750] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3561.715130] LustreError: 128586:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3561.719438] LustreError: 128586:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3561.802846] Lustre: server umount lustre-OST0000 complete [ 3567.787787] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:02:38 (1766415758) [ 3572.821880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.160154] LustreError: 129257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.214452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.108582] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3583.127475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3586.706535] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3590.398645] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3593.115662] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3598.760537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3604.272716] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3609.018863] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3624.930215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.934765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.941760] Lustre: Skipped 1 previous similar message [ 3629.538303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3629.547113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3630.748604] LustreError: 131524:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3630.752334] LustreError: 131524:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3630.827919] Lustre: server umount lustre-MDT0000 complete [ 3633.619949] LustreError: 131526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415824 with bad export cookie 3658737596542625023 [ 3633.624830] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.637458] LustreError: 131526:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3633.871721] Lustre: server umount lustre-MDT0001 complete [ 3645.835949] Lustre: server umount lustre-OST0000 complete [ 3650.934980] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:04:01 (1766415841) [ 3654.690590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3654.966942] LustreError: 132599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.980429] LustreError: 132599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3655.013300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.016464] Lustre: Skipped 2 previous similar messages [ 3657.310648] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3662.283251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3664.541384] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3667.921070] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3669.941634] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3673.697477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3676.511533] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3679.272466] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3680.046094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3681.240647] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3681.371306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3683.517508] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3687.773525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3690.471157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3690.473956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.483968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3690.486954] Lustre: Skipped 5 previous similar messages [ 3692.521496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.964143] LustreError: 135369:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3694.966798] LustreError: 135369:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3695.024244] Lustre: server umount lustre-OST0000 complete [ 3703.301427] Lustre: server umount lustre-MDT0000 complete [ 3705.609630] LustreError: 132578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415896 with bad export cookie 3658737596542626087 [ 3705.614659] LustreError: 132578:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3705.621363] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3705.883110] Lustre: server umount lustre-MDT0001 complete [ 3711.166287] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:05:01 (1766415901) [ 3715.893501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3731.424621] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3736.545090] LustreError: 136415:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 3762.208496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.213959] Lustre: Skipped 2 previous similar messages [ 3765.527054] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3768.340279] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3772.280408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.530722] LustreError: 136442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.537921] LustreError: 136442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3772.617321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3774.925243] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3780.344819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3782.574281] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3785.349853] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3787.171504] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3789.449145] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3789.587330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3791.879705] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3792.057314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3795.939341] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.943341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3795.944525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3795.951911] Lustre: Skipped 4 previous similar messages [ 3795.952454] Lustre: Skipped 6 previous similar messages [ 3799.375247] LustreError: 139189:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3799.378191] LustreError: 139189:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3799.426848] Lustre: server umount lustre-MDT0000 complete [ 3801.488881] LustreError: 136422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766415992 with bad export cookie 3658737596542627102 [ 3801.492499] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.500093] LustreError: 136422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3801.679271] Lustre: server umount lustre-MDT0001 complete [ 3814.080651] Lustre: server umount lustre-OST0000 complete [ 3818.999167] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:06:49 (1766416009) [ 3822.907927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3838.432575] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3843.552468] LustreError: 140237:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 3872.415893] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3875.232495] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3878.891365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.130629] LustreError: 140264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.144466] LustreError: 140264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3879.217517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3881.040944] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3885.400560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3887.426909] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3890.321906] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3892.289388] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3895.457970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3895.508416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3895.589340] Lustre: lustre-OST0001: new disk, initializing [ 3895.592921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3895.634635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.637368] Lustre: Skipped 5 previous similar messages [ 3898.319282] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3900.933272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3900.938342] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3900.986860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3902.626879] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3904.696513] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3904.789427] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3906.612331] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3906.720494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3910.112944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3910.116885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.122268] Lustre: Skipped 1 previous similar message [ 3910.125824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3910.128556] Lustre: Skipped 2 previous similar messages [ 3922.400268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3922.424139] LustreError: 143967:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3922.427159] LustreError: 143967:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3922.451364] Lustre: server umount lustre-OST0000 complete [ 3926.502422] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.506247] LustreError: Skipped 1 previous similar message [ 3930.477279] Lustre: server umount lustre-OST0001 complete [ 3938.500817] Lustre: server umount lustre-MDT0000 complete [ 3940.441664] LustreError: 142785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416131 with bad export cookie 3658737596542628047 [ 3940.443570] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.447261] LustreError: 142785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3940.631152] Lustre: server umount lustre-MDT0001 complete [ 3943.658667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3945.474144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3947.167809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3948.882868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3951.890120] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:09:02 (1766416142) [ 3952.766893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3953.572396] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3954.552873] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:09:05 (1766416145) [ 3958.750700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.919423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3958.937788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3961.677278] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3966.844923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3966.922995] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3966.950895] Lustre: 146210:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3969.493868] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3972.666908] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3974.646995] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3978.841130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3978.965306] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3980.139919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3982.683611] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 3985.774919] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3987.839946] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3987.959191] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3990.108033] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3990.290845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3995.617575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.618519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.621134] LustreError: Skipped 2 previous similar messages [ 3995.622694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.625988] Lustre: Skipped 6 previous similar messages [ 3995.630029] Lustre: Skipped 10 previous similar messages [ 3997.540303] Lustre: server umount lustre-OST0000 complete [ 4013.114419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4015.952201] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4016.422466] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4016.432555] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4016.434603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4018.452185] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4020.543043] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4020.653845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4022.333966] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4022.421978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4024.082708] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a52c84ac800.ost_server_uuid 50 [ 4024.759757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a52c84ac800.ost_server_uuid in FULL state after 0 sec [ 4024.839067] LustreError: 146215:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.25@tcp) returned error from blocking AST (req@00000000b0698a10 x1852220938337280 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f7abc0c3/0x32c67275904a3a30 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.25@tcp remote: 0xc1b6fbfe2568c7e5 expref: 5 pid: 146217 timeout: 4124 lvb_type: 0 [ 4024.853183] LustreError: lustre-MDT0000: A client on nid 192.168.201.25@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4024.857257] LustreError: 146205:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.25@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f7abc0c3/0x32c67275904a3a30 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.25@tcp remote: 0xc1b6fbfe2568c7e5 expref: 6 pid: 146217 timeout: 0 lvb_type: 0 [ 4032.683037] Lustre: server umount lustre-OST0000 complete [ 4049.376582] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4049.479597] Lustre: server umount lustre-MDT0000 complete [ 4051.434474] LustreError: 147848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416242 with bad export cookie 3658737596542629237 [ 4051.440086] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4051.440477] LustreError: 147848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4057.604953] LustreError: 151159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4057.608537] LustreError: 151159:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 4057.689117] Lustre: server umount lustre-MDT0001 complete [ 4062.211733] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4063.492710] Key type lgssc unregistered [ 4063.665463] LNet: 151692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4063.669857] LNetError: 151692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4063.678381] LNet: Removed LNI 192.168.201.125@tcp [ 4064.055404] Key type .llcrypt unregistered [ 4064.057116] Key type ._llcrypt unregistered [ 4076.088717] Key type ._llcrypt registered [ 4076.090760] Key type .llcrypt registered [ 4076.144471] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4076.590345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4076.627797] alg: No test for adler32 (adler32-zlib) [ 4077.518926] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4077.624667] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4079.216546] Key type lgssc registered [ 4079.682308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4083.177401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4083.189429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4084.432559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.160674] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4089.731973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4089.874459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4091.521924] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4093.883958] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4095.708821] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4098.951373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4099.143181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.558554] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4103.735850] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4109.418779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4112.514842] Lustre: Failing over lustre-MDT0000 [ 4112.686255] LustreError: 155311:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4112.735662] Lustre: server umount lustre-MDT0000 complete [ 4114.914213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.917662] LustreError: 153772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.922309] Lustre: Skipped 2 previous similar messages [ 4114.931580] LustreError: 153772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4120.033083] LustreError: 153064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.045896] LustreError: 153064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4124.005493] LustreError: 153783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.25@tcp arrived at 1766416315 with bad export cookie 13354731539416600447 [ 4124.013743] LustreError: 153783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4125.154574] LustreError: 153058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4125.177268] LustreError: 153058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4127.500572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.568349] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4127.693146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.725849] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4129.538525] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4130.651443] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4130.656764] Lustre: lustre-MDT0000: Denying connection for new client d9f020da-feef-4bbf-98df-f01c4c8073bb (at 192.168.201.25@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4132.844298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4137.955630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.956555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4137.961310] Lustre: Skipped 1 previous similar message [ 4137.965426] Lustre: Skipped 1 previous similar message [ 4143.076792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4143.080130] Lustre: Skipped 1 previous similar message [ 4143.663140] LustreError: 156245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4143.665246] LustreError: 156245:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4143.718217] Lustre: server umount lustre-OST0000 complete [ 4145.287715] LustreError: 156446:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4145.292118] Lustre: 155813:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4145.296429] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4145.306737] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4145.313031] LustreError: 155813:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000fc75ec8 x1852221704669184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4145.323197] LustreError: 155813:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4145.328480] LustreError: 155813:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4145.333813] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 4145.420746] LustreError: 156446:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4145.422838] LustreError: 156446:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4145.449303] Lustre: server umount lustre-MDT0000 complete [ 4146.920784] LustreError: 153783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416338 with bad export cookie 13354731539416600804 [ 4146.925598] LustreError: 152373:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4146.925658] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4146.925741] LustreError: 153783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4153.030620] LustreError: 156647:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4153.034884] LustreError: 156647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4153.106852] Lustre: server umount lustre-MDT0001 complete [ 4157.511355] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4158.721323] Key type lgssc unregistered [ 4158.865544] LNet: 157179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4158.869815] LNetError: 157179:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4158.877350] LNet: Removed LNI 192.168.201.125@tcp [ 4159.263334] Key type .llcrypt unregistered [ 4159.265168] Key type ._llcrypt unregistered [ 4170.950202] Key type ._llcrypt registered [ 4170.951671] Key type .llcrypt registered [ 4171.004637] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4171.450820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4171.504805] alg: No test for adler32 (adler32-zlib) [ 4172.402220] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4172.540852] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4174.160172] Key type lgssc registered [ 4174.812703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4179.089789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4179.105734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.396828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.469074] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4187.617339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4187.842127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4190.099543] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4193.205829] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4195.159894] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4198.852040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4199.032976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4201.583424] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4204.436493] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4205.167422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4210.659837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4210.660395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.670770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.705913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.709551] Lustre: Skipped 1 previous similar message [ 4214.132186] LustreError: 160796:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4214.180947] Lustre: server umount lustre-OST0000 complete [ 4217.825713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.832043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.834790] Lustre: Skipped 1 previous similar message [ 4222.945062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.947897] Lustre: Skipped 1 previous similar message [ 4228.065091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.067505] Lustre: Skipped 1 previous similar message [ 4230.112166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4230.153611] LustreError: 160997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4230.155709] LustreError: 160997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4230.191996] Lustre: server umount lustre-MDT0000 complete [ 4231.636863] LustreError: 158530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416422 with bad export cookie 3091877298742063340 [ 4231.641392] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4231.642561] LustreError: 158530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4231.722278] LustreError: 161199:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4231.725096] LustreError: 161199:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4231.786594] Lustre: server umount lustre-MDT0001 complete [ 4236.281845] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4237.488075] Key type lgssc unregistered [ 4237.633420] LNet: 161730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4237.637517] LNetError: 161730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4237.651496] LNet: Removed LNI 192.168.201.125@tcp [ 4238.036756] Key type .llcrypt unregistered [ 4238.038717] Key type ._llcrypt unregistered [ 4253.979982] Key type ._llcrypt registered [ 4253.981379] Key type .llcrypt registered [ 4254.034791] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4254.532607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4254.576720] alg: No test for adler32 (adler32-zlib) [ 4255.472538] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4255.580940] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4257.184161] Key type lgssc registered [ 4257.771530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4261.629356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4261.647488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.879221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.744496] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4268.738300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4268.942116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4270.850354] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4273.609936] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4275.547381] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4278.994149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4279.188241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.987655] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4284.514703] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4288.424449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4293.603143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.606316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.612426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4293.615261] Lustre: Skipped 1 previous similar message [ 4294.904119] LustreError: 165639:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4294.950442] Lustre: server umount lustre-OST0000 complete [ 4298.722351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.724158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.731582] Lustre: Skipped 2 previous similar messages [ 4298.780634] LustreError: 165839:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4298.782732] LustreError: 165839:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4298.816881] Lustre: server umount lustre-MDT0000 complete [ 4300.556047] LustreError: 163372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416491 with bad export cookie 8155619686067526072 [ 4300.557845] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4300.562561] LustreError: 163372:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4300.649926] LustreError: 166042:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4300.652746] LustreError: 166042:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4300.728963] Lustre: server umount lustre-MDT0001 complete [ 4305.177343] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4306.430958] Key type lgssc unregistered [ 4306.569606] LNet: 166573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4306.573691] LNetError: 166573:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4306.588441] LNet: Removed LNI 192.168.201.125@tcp [ 4306.934903] Key type .llcrypt unregistered [ 4306.937409] Key type ._llcrypt unregistered [ 4317.348478] Key type ._llcrypt registered [ 4317.350424] Key type .llcrypt registered [ 4317.406557] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4317.900208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4317.911113] alg: No test for adler32 (adler32-zlib) [ 4318.811900] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4318.921261] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4320.512166] Key type lgssc registered [ 4320.949221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4325.341076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4325.352041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.562103] Lustre: *** cfs_fail_loc=135, val=0*** [ 4326.609746] LustreError: 168054:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4326.612298] LustreError: 168054:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.201.125@tcp: cfg command failed: rc = -2 [ 4326.616305] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4326.616305] [ 4326.620526] LustreError: MGC192.168.201.125@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4326.627349] LustreError: 168013:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4326.632183] LustreError: 168013:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4326.638190] LustreError: 168013:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 4326.640814] LustreError: 168013:0:(tgt_mount.c:117:server_deregister_mount()) lustre-MDT0000 not registered [ 4326.645194] LustreError: 168013:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4326.685609] Lustre: server umount lustre-MDT0000 complete [ 4326.687641] LustreError: 168013:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4329.611602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.834274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.435183] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4334.972357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4335.149693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.762381] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4339.036931] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4340.484110] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4345.313487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4345.317573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.325682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.337612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4346.338719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.466476] LustreError: 169766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4347.470656] LustreError: 169766:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4347.504504] Lustre: server umount lustre-MDT0000 complete [ 4349.267607] LustreError: 169767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416540 with bad export cookie 17836359258733698565 [ 4349.271972] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4349.273482] LustreError: 169767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4349.368889] LustreError: 169970:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4349.373236] LustreError: 169970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4349.460073] Lustre: server umount lustre-MDT0001 complete [ 4353.869392] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4355.035840] Key type lgssc unregistered [ 4355.178468] LNet: 170501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4355.183024] LNetError: 170501:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4355.190657] LNet: Removed LNI 192.168.201.125@tcp [ 4355.532514] Key type .llcrypt unregistered [ 4355.534228] Key type ._llcrypt unregistered [ 4364.080315] Key type ._llcrypt registered [ 4364.081915] Key type .llcrypt registered [ 4364.129371] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4364.251432] Key type .llcrypt unregistered [ 4364.253112] Key type ._llcrypt unregistered [ 4371.850396] Key type ._llcrypt registered [ 4371.851573] Key type .llcrypt registered [ 4371.896898] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4372.236529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.281101] alg: No test for adler32 (adler32-zlib) [ 4373.161705] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4373.263555] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4374.856161] Key type lgssc registered [ 4375.305365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4379.007672] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4379.025221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4394.528523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4399.648224] LustreError: 172567:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.125@tcp: failed processing log, type 4: rc = -110 [ 4426.272665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.328296] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4430.484439] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4433.126723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.341586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.365098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4434.880525] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4438.236570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4438.412962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.951647] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4441.996689] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4443.359455] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4444.453053] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4448.738117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4448.739089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.741461] LustreError: Skipped 1 previous similar message [ 4448.746349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.749260] Lustre: Skipped 1 previous similar message [ 4451.810079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4451.812608] Lustre: Skipped 1 previous similar message [ 4454.190898] LustreError: 175145:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4454.231854] Lustre: server umount lustre-OST0000 complete [ 4456.929595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4456.930263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4456.934169] Lustre: Skipped 2 previous similar messages [ 4456.936807] Lustre: Skipped 1 previous similar message [ 4461.695308] LustreError: 175347:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4461.698185] LustreError: 175347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4461.732169] Lustre: server umount lustre-MDT0000 complete [ 4462.048861] LustreError: 173495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4462.055114] LustreError: 173495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4463.101375] LustreError: 172575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416654 with bad export cookie 13649468149418692604 [ 4463.103809] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4463.111226] LustreError: 172575:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4463.172175] LustreError: 175549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4463.175480] LustreError: 175549:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4463.234184] Lustre: server umount lustre-MDT0001 complete [ 4467.089226] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4468.188740] Key type lgssc unregistered [ 4468.318393] LNet: 176080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4468.321467] LNetError: 176080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4468.330327] LNet: Removed LNI 192.168.201.125@tcp [ 4468.662313] Key type .llcrypt unregistered [ 4468.663692] Key type ._llcrypt unregistered [ 4478.197948] Key type ._llcrypt registered [ 4478.199532] Key type .llcrypt registered [ 4478.243380] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4478.778381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4478.791845] alg: No test for adler32 (adler32-zlib) [ 4479.658515] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4479.752181] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4481.336187] Key type lgssc registered [ 4481.734146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4484.493564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4484.500260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.658192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.010258] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4490.021603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4490.162410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4491.547062] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4493.542546] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4494.843627] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4497.351970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4497.474079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.466313] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4501.388940] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4509.799030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4513.448830] Lustre: Failing over lustre-MDT0000 [ 4513.523467] LustreError: 179767:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4513.557785] Lustre: server umount lustre-MDT0000 complete [ 4515.299524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.300176] LustreError: 177447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.304971] Lustre: Skipped 2 previous similar messages [ 4515.313359] LustreError: 177447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4517.156392] LustreError: 179530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.417031] LustreError: 178173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.421984] LustreError: 178173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4525.538374] LustreError: 178173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.543966] LustreError: 178173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4526.716873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.756093] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.840253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.875401] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4527.386415] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4528.180810] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4529.785886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4532.198612] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4532.206256] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4532.225553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4533.118800] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4547.040755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4547.043412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.048443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4547.555941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4547.556295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.700707] LustreError: 181024:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4550.702648] LustreError: 181024:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4550.742458] Lustre: server umount lustre-OST0000 complete [ 4552.673485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.673935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4552.674238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.678766] Lustre: Skipped 2 previous similar messages [ 4557.794829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4557.797257] Lustre: Skipped 2 previous similar messages [ 4558.207512] LustreError: 181225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4558.209661] LustreError: 181225:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4558.238194] Lustre: server umount lustre-MDT0000 complete [ 4559.633494] LustreError: 177430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416750 with bad export cookie 16906195412267807904 [ 4559.636128] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.638031] LustreError: 177430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4559.764388] Lustre: server umount lustre-MDT0001 complete [ 4563.578230] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4564.699842] Key type lgssc unregistered [ 4564.854379] LNet: 181957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4564.856987] LNetError: 181957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4564.866586] LNet: Removed LNI 192.168.201.125@tcp [ 4565.203977] Key type .llcrypt unregistered [ 4565.205524] Key type ._llcrypt unregistered [ 4575.661689] Key type ._llcrypt registered [ 4575.663159] Key type .llcrypt registered [ 4575.711443] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4576.162836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4576.197928] alg: No test for adler32 (adler32-zlib) [ 4577.083567] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4577.185793] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4578.784451] Key type lgssc registered [ 4579.240450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4582.580762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4582.593889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.839764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4585.530932] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4589.268721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4589.448432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4591.176251] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4593.549917] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4595.013568] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4598.342423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4598.585693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.407403] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4603.915781] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4608.810041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4610.841521] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4613.399440] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4618.469293] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4629.475137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.477265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4629.481559] Lustre: Skipped 1 previous similar message [ 4629.482202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4629.484616] Lustre: Skipped 1 previous similar message [ 4634.098163] LustreError: 185745:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4634.153793] Lustre: server umount lustre-OST0000 complete [ 4639.713789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.715301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.719453] Lustre: Skipped 1 previous similar message [ 4641.985957] LustreError: 185946:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4641.988072] LustreError: 185946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4642.019370] Lustre: server umount lustre-MDT0000 complete [ 4643.596535] LustreError: 185534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416834 with bad export cookie 15960432287072482879 [ 4643.599755] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4643.603861] LustreError: 185534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4643.675749] LustreError: 186147:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4643.678681] LustreError: 186147:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4643.742210] Lustre: server umount lustre-MDT0001 complete [ 4647.870732] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4649.279404] Key type lgssc unregistered [ 4649.436498] LNet: 186678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4649.440710] LNetError: 186678:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4649.449614] LNet: Removed LNI 192.168.201.125@tcp [ 4649.844074] Key type .llcrypt unregistered [ 4649.845898] Key type ._llcrypt unregistered [ 4660.516882] Key type ._llcrypt registered [ 4660.518857] Key type .llcrypt registered [ 4660.574719] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4661.118933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4661.169638] alg: No test for adler32 (adler32-zlib) [ 4662.103837] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4662.226182] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4663.848659] Key type lgssc registered [ 4664.378944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4668.053802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4668.063686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.317071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.018439] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4674.643699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4674.802255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4676.478494] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4678.808945] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4680.193419] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4683.062379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4683.245359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4685.682315] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4687.888825] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4694.513585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4700.632260] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4711.552742] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4721.300518] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4729.325766] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4740.579710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4740.580057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.589197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4740.595687] Lustre: Skipped 1 previous similar message [ 4745.698531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4745.701008] Lustre: Skipped 1 previous similar message [ 4750.818932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4750.822585] Lustre: Skipped 1 previous similar message [ 4752.864134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4752.881172] LustreError: 192379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4752.921820] Lustre: server umount lustre-OST0000 complete [ 4755.937537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4755.938281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4755.942965] Lustre: Skipped 2 previous similar messages [ 4760.511733] LustreError: 192581:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4760.514372] LustreError: 192581:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4760.544443] Lustre: server umount lustre-MDT0000 complete [ 4761.057162] LustreError: 188048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.063798] LustreError: 188048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4761.985483] LustreError: 189670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766416953 with bad export cookie 2300873228289559922 [ 4761.989328] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.990257] LustreError: 189670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4762.069352] LustreError: 192782:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4762.071637] LustreError: 192782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4762.132191] Lustre: server umount lustre-MDT0001 complete [ 4766.179344] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4767.383354] Key type lgssc unregistered [ 4767.524383] LNet: 193313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4767.527176] LNetError: 193313:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4767.539355] LNet: Removed LNI 192.168.201.125@tcp [ 4767.875738] Key type .llcrypt unregistered [ 4767.877651] Key type ._llcrypt unregistered [ 4777.621503] Key type ._llcrypt registered [ 4777.623094] Key type .llcrypt registered [ 4777.664268] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4777.999021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4778.030258] alg: No test for adler32 (adler32-zlib) [ 4778.896799] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4778.993601] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4780.584195] Key type lgssc registered [ 4780.988064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4783.776953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4783.785481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.957315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.378963] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4789.455024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4789.592614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4791.053443] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4792.987267] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4794.309764] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4796.822712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4796.968365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.113857] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4801.154367] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.282805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4815.331353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.332333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4815.332649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4815.335996] Lustre: Skipped 1 previous similar message [ 4817.267163] LustreError: 197101:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4817.321617] Lustre: server umount lustre-OST0000 complete [ 4820.449791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.449834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4820.450453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4820.450457] Lustre: Skipped 1 previous similar message [ 4820.454962] Lustre: Skipped 1 previous similar message [ 4820.799393] LustreError: 197301:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4820.802615] LustreError: 197301:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4820.833371] Lustre: server umount lustre-MDT0000 complete [ 4822.215501] LustreError: 194664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766417013 with bad export cookie 13235257063419597511 [ 4822.217515] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4822.219548] LustreError: 194664:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4822.288290] LustreError: 197503:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4822.291103] LustreError: 197503:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4822.347660] Lustre: server umount lustre-MDT0001 complete [ 4825.978923] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4827.021562] Key type lgssc unregistered [ 4827.159506] LNet: 198034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4827.162773] LNetError: 198034:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4827.172326] LNet: Removed LNI 192.168.201.125@tcp [ 4827.471578] Key type .llcrypt unregistered [ 4827.472988] Key type ._llcrypt unregistered [ 4834.637544] Key type ._llcrypt registered [ 4834.639063] Key type .llcrypt registered [ 4834.679469] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4835.045109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4835.079083] alg: No test for adler32 (adler32-zlib) [ 4835.931089] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4836.020637] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4837.600145] Key type lgssc registered [ 4837.966283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4840.832319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4840.842317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.045075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4843.635102] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4847.248466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4847.428556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.093231] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4851.470489] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4853.065430] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4855.996369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4856.162914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4858.532552] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4860.908102] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4867.431716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4877.794548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4877.795433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4877.799614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4877.802111] LustreError: Skipped 1 previous similar message [ 4877.807272] Lustre: Skipped 1 previous similar message [ 4881.655903] LustreError: 201672:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4881.718364] Lustre: server umount lustre-OST0000 complete [ 4883.428346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4883.431736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.441469] Lustre: Skipped 1 previous similar message [ 4883.443803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.033209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4888.033788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4889.413732] LustreError: 201873:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4889.416770] LustreError: 201873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4889.460606] Lustre: server umount lustre-MDT0000 complete [ 4891.024566] LustreError: 201586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766417082 with bad export cookie 11360953487996681103 [ 4891.028240] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4891.030297] LustreError: 201586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4891.105207] LustreError: 202076:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4891.107942] LustreError: 202076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4891.165156] Lustre: server umount lustre-MDT0001 complete [ 4895.238813] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 4896.391923] Key type lgssc unregistered [ 4896.534409] LNet: 202607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4896.537523] LNetError: 202607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4896.544368] LNet: Removed LNI 192.168.201.125@tcp [ 4896.881716] Key type .llcrypt unregistered [ 4896.883317] Key type ._llcrypt unregistered [ 4905.292965] Key type ._llcrypt registered [ 4905.294587] Key type .llcrypt registered [ 4905.338237] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 4905.689443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4905.757848] alg: No test for adler32 (adler32-zlib) [ 4906.624212] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 4906.715337] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 4908.304197] Key type lgssc registered [ 4908.752891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4911.883958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4911.893041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.107618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.668441] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4918.711852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4918.927316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4920.734935] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4923.064341] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4924.603958] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4927.427129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4927.604571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4929.964689] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4932.203387] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4937.834269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4945.973918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4946.016547] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4946.140584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4948.942347] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 4951.349357] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4951.991651] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4962.730182] Lustre: Permanently deactivating lustre-OST0001 [ 4962.737165] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4971.014160] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4976.361851] Lustre: Permanently reactivating lustre-OST0001 [ 4976.367178] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4976.370266] Lustre: Skipped 2 previous similar messages [ 4977.276442] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.282151] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4977.285489] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4977.291118] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4977.304867] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4987.873893] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4987.874274] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.876845] LustreError: Skipped 1 previous similar message [ 4987.881482] Lustre: Skipped 1 previous similar message [ 4987.887553] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4987.891571] Lustre: Skipped 1 previous similar message [ 4989.413179] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4989.415662] Lustre: Skipped 1 previous similar message [ 4992.051919] LustreError: 207347:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4992.073575] Lustre: server umount lustre-OST0001 complete [ 4994.532912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4994.534667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.535737] LustreError: 205424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.535755] LustreError: 205424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4994.537119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4994.542784] Lustre: Skipped 1 previous similar message [ 4994.557661] Lustre: Skipped 1 previous similar message [ 4999.653904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4999.657833] Lustre: Skipped 1 previous similar message [ 4999.660089] LustreError: 205427:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4999.677575] LustreError: 205427:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5004.773044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5004.773490] LustreError: 205424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.779272] Lustre: Skipped 1 previous similar message [ 5004.787319] LustreError: 205424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5007.840150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5007.864424] LustreError: 207549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5007.866485] LustreError: 207549:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5007.912920] Lustre: server umount lustre-OST0000 complete [ 5009.890077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5015.009420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.014282] Lustre: Skipped 3 previous similar messages [ 5015.494757] LustreError: 207751:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5015.497465] LustreError: 207751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5015.527050] Lustre: server umount lustre-MDT0000 complete [ 5017.111114] LustreError: 203812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766417208 with bad export cookie 7532156545914600790 [ 5017.115893] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.116517] LustreError: 203812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5017.250791] Lustre: server umount lustre-MDT0001 complete [ 5021.380769] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5022.563560] Key type lgssc unregistered [ 5022.700846] LNet: 208485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5022.704786] LNetError: 208485:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5022.716365] LNet: Removed LNI 192.168.201.125@tcp [ 5023.080738] Key type .llcrypt unregistered [ 5023.082061] Key type ._llcrypt unregistered [ 5036.078206] Key type ._llcrypt registered [ 5036.079634] Key type .llcrypt registered [ 5036.127938] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5042.719436] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5043.149202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5043.165775] alg: No test for adler32 (adler32-zlib) [ 5044.028678] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 5044.122199] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5045.712178] Key type lgssc registered [ 5046.208903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5049.642075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5052.125832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5054.381055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5058.142481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5061.909993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5061.929750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5061.936414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.044285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5063.061166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5063.106485] Lustre: lustre-MDT0000: new disk, initializing [ 5063.143459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.152442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5064.701099] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5069.217839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5069.240210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5069.278908] Lustre: 212404:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5069.306127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5069.309127] Lustre: Skipped 1 previous similar message [ 5069.348516] Lustre: lustre-MDT0001: new disk, initializing [ 5069.383756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.397701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5069.402143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5070.971439] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5074.525967] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5076.067209] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5078.651080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5078.674719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5078.777432] Lustre: lustre-OST0000: new disk, initializing [ 5078.780707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5078.816430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5079.487539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5079.492332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5079.545255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5080.936389] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5084.227107] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.718646] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5085.792386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5087.236305] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5087.319404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5089.762155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5089.764252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.765177] LustreError: Skipped 1 previous similar message [ 5089.769918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.772498] Lustre: Skipped 1 previous similar message [ 5094.063148] LustreError: 215344:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5094.105161] Lustre: server umount lustre-OST0000 complete [ 5100.002033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.002043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5100.003889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.006785] Lustre: Skipped 2 previous similar messages [ 5101.570084] LustreError: 215546:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5101.571951] LustreError: 215546:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5101.602699] Lustre: server umount lustre-MDT0000 complete [ 5102.933320] LustreError: 212395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766417294 with bad export cookie 16060494600106223183 [ 5102.935931] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.937572] LustreError: 212395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5103.005568] LustreError: 215748:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5103.007962] LustreError: 215748:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5103.064411] Lustre: server umount lustre-MDT0001 complete [ 5105.983701] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 10:28:17 (1766417297) [ 5108.416233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5108.620467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.994666] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5113.022220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5113.151299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5114.504916] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5116.339121] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5117.518357] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5119.842152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5122.030590] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5123.881548] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5135.267568] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5139.758447] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5144.288177] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5152.847253] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5165.974065] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5165.977651] Lustre: Skipped 1 previous similar message [ 5190.684349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5190.687117] Lustre: Skipped 2 previous similar messages [ 5228.604883] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5228.607145] Lustre: Skipped 4 previous similar messages [ 5292.001164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5292.001235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.004017] LustreError: Skipped 1 previous similar message [ 5292.008332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5292.011406] Lustre: Skipped 2 previous similar messages [ 5295.074432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5296.301177] LustreError: 219922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5296.303424] LustreError: 219922:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5296.341613] Lustre: server umount lustre-OST0000 complete [ 5300.193926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.194467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5300.200356] Lustre: Skipped 2 previous similar messages [ 5300.202108] Lustre: Skipped 1 previous similar message [ 5303.742983] LustreError: 220124:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5303.745781] LustreError: 220124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5303.778898] Lustre: server umount lustre-MDT0000 complete [ 5305.036985] LustreError: 218686:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766417496 with bad export cookie 16060494600106224051 [ 5305.040109] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5305.042932] LustreError: 218686:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5305.163859] Lustre: server umount lustre-MDT0001 complete [ 5308.592283] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5309.598215] Key type lgssc unregistered [ 5309.724286] LNet: 220856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5309.727321] LNetError: 220856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5309.736283] LNet: Removed LNI 192.168.201.125@tcp [ 5310.056556] Key type .llcrypt unregistered [ 5310.057983] Key type ._llcrypt unregistered [ 5319.113821] Key type ._llcrypt registered [ 5319.114883] Key type .llcrypt registered [ 5319.159794] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5319.634651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5319.651576] alg: No test for adler32 (adler32-zlib) [ 5320.502192] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 5320.586530] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5322.168142] Key type lgssc registered [ 5322.509988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5325.004393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5325.008928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.166701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5327.389120] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5330.099039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5330.223527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5331.478326] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5333.278332] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5334.440400] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5336.595250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5336.705845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.431263] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_default_debug -1 all [ 5340.070957] Lustre: DEBUG MARKER: oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5360.609115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5360.611066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.614422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.635213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5361.636054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.636407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.644899] Lustre: Skipped 1 previous similar message [ 5365.748117] LustreError: 224646:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5365.779794] Lustre: server umount lustre-OST0000 complete [ 5371.360994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5371.363307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5371.367122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.119456] LustreError: 224851:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5373.121401] LustreError: 224851:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5373.151442] Lustre: server umount lustre-MDT0000 complete [ 5374.378173] LustreError: 224559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1766417565 with bad export cookie 9972884120382795976 [ 5374.381066] LustreError: 224559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5374.381209] LustreError: MGC192.168.201.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5374.454496] LustreError: 225052:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5374.456578] LustreError: 225052:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5374.506223] Lustre: server umount lustre-MDT0001 complete [ 5378.087228] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5379.186284] Key type lgssc unregistered [ 5379.319524] LNet: 225582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5379.322186] LNetError: 225582:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5379.334458] LNet: Removed LNI 192.168.201.125@tcp [ 5379.660680] Key type .llcrypt unregistered [ 5379.662448] Key type ._llcrypt unregistered [ 5388.158341] Key type ._llcrypt registered [ 5388.159775] Key type .llcrypt registered [ 5388.206691] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5388.706404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5388.721783] alg: No test for adler32 (adler32-zlib) [ 5389.597160] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 5389.694840] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5391.288161] Key type lgssc registered [ 5391.690100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5657.937360] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5659.084551] Key type lgssc unregistered [ 5659.224456] LNet: 227545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5659.226559] LNetError: 227545:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5659.234412] LNet: Removed LNI 192.168.201.125@tcp [ 5659.579242] Key type .llcrypt unregistered [ 5659.581032] Key type ._llcrypt unregistered [ 5668.177224] Key type ._llcrypt registered [ 5668.179206] Key type .llcrypt registered [ 5668.221450] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing set_hostid [ 5674.837618] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing load_modules_local [ 5675.174912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5675.181400] alg: No test for adler32 (adler32-zlib) [ 5676.038731] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 5676.125878] LNet: Added LNI 192.168.201.125@tcp [8/256/0/180] [ 5677.712178] Key type lgssc registered [ 5678.141860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5681.306734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5683.675569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.866660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5688.088451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5688.655360] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5436 sec ========== 10:37:59 (1766417879) [ 5689.271580] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:38:00 (1766417880) === [ 5690.363121] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:38:01 (1766417881) === [ 5704.083336] Lustre: DEBUG MARKER: oleg125-server.virtnet: executing unload_modules_local [ 5705.207249] Key type lgssc unregistered [ 5705.350434] LNet: 232706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5705.354734] LNetError: 232706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5705.370375] LNet: Removed LNI 192.168.201.125@tcp [ 5705.710859] Key type .llcrypt unregistered [ 5705.712072] Key type ._llcrypt unregistered