[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511387025 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002490] x2apic enabled [ 0.003015] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.009131] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.019724] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023689] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] *** VALIDATE tmpfs *** [ 0.029175] *** VALIDATE proc *** [ 0.030216] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.032224] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035022] Spectre V2 : User space: Vulnerable [ 0.036005] Speculative Store Bypass: Vulnerable [ 0.039659] debug: unmapping init [mem 0xffffffff96a59000-0xffffffff96a60fff] [ 0.041157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042698] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043018] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045007] ... generic registers: 4 [ 0.046008] ... value mask: 0000ffffffffffff [ 0.047008] ... max period: 00007fffffffffff [ 0.048007] ... fixed-purpose events: 3 [ 0.048930] ... event mask: 000000070000000f [ 0.049289] rcu: Hierarchical SRCU implementation. [ 0.051447] smp: Bringing up secondary CPUs ... [ 0.052519] x86: Booting SMP configuration: [ 0.053015] .... node #0, CPUs: #1 #2 #3 [ 0.057738] smp: Brought up 1 node, 4 CPUs [ 0.059018] smpboot: Max logical packages: 1 [ 0.060023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.230021] node 0 deferred pages initialised in 168ms [ 0.234168] devtmpfs: initialized [ 0.235283] x86/mm: Memory block size: 128MB [ 0.237794] gcov: version magic: 0x41383552 [ 0.240295] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244313] pinctrl core: initialized pinctrl subsystem [ 0.246141] [ 0.246593] ************************************************************* [ 0.248008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250008] ** ** [ 0.251010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.253008] ** ** [ 0.255007] ** This means that this kernel is built to expose internal ** [ 0.256006] ** IOMMU data structures, which may compromise security on ** [ 0.258007] ** your system. ** [ 0.260008] ** ** [ 0.262007] ** If you see this message and you are not debugging the ** [ 0.263006] ** kernel, report this immediately to your vendor! ** [ 0.265010] ** ** [ 0.267011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268009] ************************************************************* [ 0.269902] NET: Registered protocol family 16 [ 0.271497] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.273073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.275064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.278164] cpuidle: using governor menu [ 0.280513] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.282543] PCI: Using configuration type 1 for base access [ 0.284151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295332] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.300343] cryptd: max_cpu_qlen set to 1000 [ 0.305000] ACPI: Added _OSI(Module Device) [ 0.306013] ACPI: Added _OSI(Processor Device) [ 0.307007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.312000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317648] ACPI: Interpreter enabled [ 0.318094] ACPI: PM: (supports S0 S3 S4 S5) [ 0.319012] ACPI: Using IOAPIC for interrupt routing [ 0.320133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.323425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.327000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.333019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.336085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.341462] acpiphp: Slot [2] registered [ 0.343136] acpiphp: Slot [3] registered [ 0.344084] acpiphp: Slot [4] registered [ 0.345088] acpiphp: Slot [5] registered [ 0.347125] acpiphp: Slot [6] registered [ 0.348148] acpiphp: Slot [7] registered [ 0.350088] acpiphp: Slot [8] registered [ 0.351116] acpiphp: Slot [9] registered [ 0.353087] acpiphp: Slot [10] registered [ 0.354089] acpiphp: Slot [11] registered [ 0.356059] acpiphp: Slot [12] registered [ 0.357150] acpiphp: Slot [13] registered [ 0.359069] acpiphp: Slot [14] registered [ 0.360071] acpiphp: Slot [15] registered [ 0.361092] acpiphp: Slot [16] registered [ 0.362000] acpiphp: Slot [17] registered [ 0.362000] acpiphp: Slot [18] registered [ 0.362085] acpiphp: Slot [19] registered [ 0.363000] acpiphp: Slot [20] registered [ 0.365062] acpiphp: Slot [21] registered [ 0.366064] acpiphp: Slot [22] registered [ 0.367092] acpiphp: Slot [23] registered [ 0.369086] acpiphp: Slot [24] registered [ 0.370063] acpiphp: Slot [25] registered [ 0.371088] acpiphp: Slot [26] registered [ 0.372063] acpiphp: Slot [27] registered [ 0.373055] acpiphp: Slot [28] registered [ 0.375061] acpiphp: Slot [29] registered [ 0.376058] acpiphp: Slot [30] registered [ 0.377083] acpiphp: Slot [31] registered [ 0.378050] PCI host bridge to bus 0000:00 [ 0.380017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.382024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.386027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.388011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.390010] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.392017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.393185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.397252] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.401337] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.409022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.416769] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.419014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.420016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.424025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.425566] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.428641] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.430040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.432613] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.437015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448015] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.453013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.499070] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.521022] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.534157] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.538018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.545014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.560018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.568108] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.579000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.589016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.600491] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.610015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.621022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.640020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.648000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.656022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.666021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.677000] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.686461] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.694018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.701017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.723018] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.732073] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.733277] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.735481] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.736334] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.739204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.743834] iommu: Default domain type: Passthrough [ 0.745435] SCSI subsystem initialized [ 0.746133] ACPI: bus type USB registered [ 0.747085] usbcore: registered new interface driver usbfs [ 0.749064] usbcore: registered new interface driver hub [ 0.751077] usbcore: registered new device driver usb [ 0.753170] pps_core: LinuxPPS API ver. 1 registered [ 0.754009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.757053] PTP clock support registered [ 0.761037] EDAC MC: Ver: 3.0.0 [ 0.763608] PCI: Using ACPI for IRQ routing [ 0.764000] NetLabel: Initializing [ 0.764000] NetLabel: domain hash size = 128 [ 0.765010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.767096] NetLabel: unlabeled traffic allowed by default [ 0.769303] vgaarb: loaded [ 0.771839] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.773008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.775000] clocksource: Switched to clocksource kvm-clock [ 0.894589] VFS: Disk quotas dquot_6.6.0 [ 0.895897] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.898225] *** VALIDATE ramfs *** [ 0.899415] *** VALIDATE hugetlbfs *** [ 0.900864] pnp: PnP ACPI init [ 0.903234] pnp: PnP ACPI: found 6 devices [ 0.920943] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.924313] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.926355] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.928546] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.930903] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.933238] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.935916] NET: Registered protocol family 2 [ 0.938258] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.942913] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.946573] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951630] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955052] TCP: Hash tables configured (established 65536 bind 65536) [ 0.958167] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.961330] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964012] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967153] NET: Registered protocol family 1 [ 0.970352] RPC: Registered named UNIX socket transport module. [ 0.972544] RPC: Registered udp transport module. [ 0.974276] RPC: Registered tcp transport module. [ 0.975959] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.978065] NET: Registered protocol family 44 [ 0.979542] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.981536] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.983510] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.985680] PCI: CLS 0 bytes, default 64 [ 0.987786] Unpacking initramfs... [ 2.668518] debug: unmapping init [mem 0xffff97e67cc54000-0xffff97e67ffbffff] [ 2.674285] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.676214] software IO TLB: mapped [mem 0x00000000a2200000-0x00000000a6200000] (64MB) [ 2.679562] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.233776] Initialise system trusted keyrings [ 3.235222] Key type blacklist registered [ 3.237404] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.248263] zbud: loaded [ 3.251207] *** VALIDATE nfs *** [ 3.252360] *** VALIDATE nfs4 *** [ 3.259407] pstore: using deflate compression [ 3.262887] Platform Keyring initialized [ 3.427224] NET: Registered protocol family 38 [ 3.428724] Key type asymmetric registered [ 3.447841] Asymmetric key parser 'x509' registered [ 3.449398] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.463390] io scheduler mq-deadline registered [ 3.464899] io scheduler kyber registered [ 3.466192] io scheduler bfq registered [ 3.474718] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.478460] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.480816] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.483139] ACPI: Power Button [PWRF] [ 3.587101] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.689921] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.912977] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.017531] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.226783] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.253968] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.298263] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.308128] Non-volatile memory driver v1.3 [ 4.309356] Linux agpgart interface v0.103 [ 4.337612] virtio_blk virtio1: [vda] 133240 512-byte logical blocks (68.2 MB/65.1 MiB) [ 4.341468] vda: detected capacity change from 0 to 68218880 [ 4.364474] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.369184] vdb: detected capacity change from 0 to 1073741824 [ 4.396909] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.400254] vdc: detected capacity change from 0 to 2621440000 [ 4.423095] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.426703] vdd: detected capacity change from 0 to 2621440000 [ 4.453979] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.461287] vde: detected capacity change from 0 to 4294967296 [ 4.484387] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.488335] vdf: detected capacity change from 0 to 4294967296 [ 4.496373] libphy: Fixed MDIO Bus: probed [ 4.502908] usbcore: registered new interface driver usbserial_generic [ 4.505423] usbserial: USB Serial support registered for generic [ 4.510759] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.519147] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.520752] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.526929] mousedev: PS/2 mouse device common for all mice [ 4.529327] rtc_cmos 00:05: RTC can wake from S4 [ 4.531264] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.535326] rtc_cmos 00:05: registered as rtc0 [ 4.536803] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.542034] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.545986] intel_pstate: CPU model not supported [ 4.549132] hid: raw HID events driver (C) Jiri Kosina [ 4.552844] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.559865] usbcore: registered new interface driver usbhid [ 4.565215] usbhid: USB HID core driver [ 4.566711] drop_monitor: Initializing network drop monitor service [ 4.568766] Initializing XFRM netlink socket [ 4.574393] NET: Registered protocol family 10 [ 4.576762] Segment Routing with IPv6 [ 4.577966] NET: Registered protocol family 17 [ 4.579581] mpls_gso: MPLS GSO support [ 4.590623] RAS: Correctable Errors collector initialized. [ 4.592557] AVX version of gcm_enc/dec engaged. [ 4.594069] AES CTR mode by8 optimization enabled [ 4.708864] sched_clock: Marking stable (4708844529, 0)->(5730273707, -1021429178) [ 4.712569] registered taskstats version 1 [ 4.714683] Loading compiled-in X.509 certificates [ 4.716537] zswap: loaded using pool lzo/zbud [ 4.750613] Key type big_key registered [ 4.764718] Key type encrypted registered [ 4.767321] ima: No TPM chip found, activating TPM-bypass! [ 4.769637] ima: Allocated hash algorithm: sha1 [ 4.771494] ima: No architecture policies found [ 4.773742] evm: Initialising EVM extended attributes: [ 4.775163] evm: security.selinux [ 4.776200] evm: security.ima [ 4.777458] evm: security.capability [ 4.778850] evm: HMAC attrs: 0x1 [ 4.786960] rtc_cmos 00:05: setting system clock to 2025-09-25 18:07:01 UTC (1758823621) [ 4.794323] debug: unmapping init [mem 0xffffffff97a03000-0xffffffff97bfffff] [ 4.797281] debug: unmapping init [mem 0xffffffff96782000-0xffffffff96a58fff] [ 4.810263] Write protecting the kernel read-only data: 28672k [ 4.814693] debug: unmapping init [mem 0xffffffff94e03000-0xffffffff94ffffff] [ 4.817454] debug: unmapping init [mem 0xffffffff95714000-0xffffffff957fffff] [ 4.865149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.872015] systemd[1]: Detected virtualization kvm. [ 4.873566] systemd[1]: Detected architecture x86-64. [ 4.875152] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.924629] systemd[1]: No hostname configured. [ 4.926069] systemd[1]: Set hostname to . [ 4.927709] random: systemd: uninitialized urandom read (16 bytes read) [ 4.929900] systemd[1]: Initializing machine ID from random generator. [ 5.168108] random: systemd: uninitialized urandom read (16 bytes read) [ 5.170505] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.190679] random: systemd: uninitialized urandom read (16 bytes read) [ 5.192794] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.199806] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.413485] device-mapper: uevent: version 1.0.3 [ 6.416142] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.594565] virtio_net virtio0 ens2: renamed from eth0 [ 7.775519] random: fast init done [ 8.093727] scsi host0: ata_piix [ 8.110971] scsi host1: ata_piix [ 8.148451] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.150712] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.293596] random: crng init done [ 12.294846] random: 7 urandom warning(s) missed due to ratelimiting [ 14.553375] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.332207] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.047272] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.838685] SELinux: Disabled at runtime. [ 17.956587] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.973868] systemd[1]: Detected virtualization kvm. [ 17.975063] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.363619] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.368888] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.377920] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.392463] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.404369] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.436264] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.479110] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ 19.687478] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.917437] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.824916] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.869794] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.170441] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.199842] EDAC sbridge: Ver: 1.1.2 [ 24.152753] Key type dns_resolver registered [ 24.551330] NFS: Registering the id_resolver key type [ 24.553247] Key type id_resolver registered [ 24.554679] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg421-server login: [ 48.546265] libcfs: loading out-of-tree module taints kernel. [ 48.557091] Key type ._llcrypt registered [ 48.558656] Key type .llcrypt registered [ 48.595649] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 55.232980] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 55.830176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.837845] alg: No test for adler32 (adler32-zlib) [ 56.840386] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 57.115900] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 58.727164] Key type lgssc registered [ 59.365700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.657461] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.676991] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.253030] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.789370] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.352326] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.480613] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 84.680533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.698603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 84.707132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.787460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 85.800922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 85.837431] Lustre: lustre-MDT0000: new disk, initializing [ 85.857767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.862674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 87.067731] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.900955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.923851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 91.949062] Lustre: 6492:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 91.959072] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 91.960497] Lustre: Skipped 1 previous similar message [ 91.994165] Lustre: lustre-MDT0001: new disk, initializing [ 92.013432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 92.020092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 92.023991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 93.215981] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.441410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.422606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.448935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 98.538850] Lustre: lustre-OST0000: new disk, initializing [ 98.541983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.559441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.468318] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.402757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.407084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 103.426610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 106.054565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.082280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.120790] Lustre: lustre-OST0001: new disk, initializing [ 106.123750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 106.143434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.023073] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.131119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 113.133944] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 113.149547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 113.860959] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.637715] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 125.689878] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing check_logdir /tmp/testlogs/ [ 127.209938] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing yml_node [ 128.696219] Lustre: DEBUG MARKER: Client: 2.16.58.75 [ 129.556517] Lustre: DEBUG MARKER: MDS: 2.16.58.75 [ 130.430771] Lustre: DEBUG MARKER: OSS: 2.16.58.75 [ 130.992713] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 25 14:09:07 EDT 2025 [ 136.711123] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 137.179166] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 148.960071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.960542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.962986] Lustre: Skipped 2 previous similar messages [ 148.965527] Lustre: Skipped 2 previous similar messages [ 153.567528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 153.569605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.574330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 159.199659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 159.201663] Lustre: Skipped 7 previous similar messages [ 163.295102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 163.326244] LustreError: 12196:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 163.352525] Lustre: server umount lustre-MDT0000 complete [ 164.689540] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823781 with bad export cookie 10428378005783887247 [ 164.691054] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 164.692741] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 169.439525] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 169.439643] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.440039] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 169.444830] LustreError: 6498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 169.447958] Lustre: Skipped 1 previous similar message [ 170.811589] LustreError: 12399:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 170.813415] LustreError: 12399:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 170.885981] Lustre: server umount lustre-MDT0001 complete [ 172.302144] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 172.303942] LustreError: 12601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 172.324835] Lustre: server umount lustre-OST0000 complete [ 173.769762] Lustre: server umount lustre-OST0001 complete [ 175.981185] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 179.416128] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 183.235037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.726838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.052737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.389186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.755772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.779916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.865838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 192.878853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 192.915940] Lustre: lustre-MDT0000: new disk, initializing [ 192.942776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.949992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.370119] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 198.704734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.724749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 198.762243] Lustre: 14980:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 198.879264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.370368] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 203.592110] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 204.944918] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 207.273204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 207.297538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 207.394279] Lustre: lustre-OST0000: new disk, initializing [ 207.395604] Lustre: Skipped 1 previous similar message [ 207.397195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.399333] Lustre: Skipped 2 previous similar messages [ 207.425820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.427695] Lustre: Skipped 1 previous similar message [ 209.209650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.211894] Lustre: Skipped 1 previous similar message [ 209.213502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.249111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.305468] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 212.419216] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 213.835806] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 213.917923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 215.322857] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 215.409178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 219.618370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 219.619836] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.620831] LustreError: Skipped 1 previous similar message [ 219.625535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 219.627935] Lustre: Skipped 2 previous similar messages [ 229.858621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 229.860542] Lustre: Skipped 3 previous similar messages [ 230.367118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 230.385057] LustreError: 17921:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 230.387100] LustreError: 17921:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 230.448113] Lustre: server umount lustre-OST0000 complete [ 234.977060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 234.977104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.984042] Lustre: Skipped 2 previous similar messages [ 246.239159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 246.287393] LustreError: 18123:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 246.290530] LustreError: 18123:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 246.331232] Lustre: server umount lustre-MDT0000 complete [ 248.442758] LustreError: 15762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823865 with bad export cookie 10428378005783888766 [ 248.446247] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 248.459051] LustreError: 15762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 248.641979] Lustre: server umount lustre-MDT0001 complete [ 250.572859] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:11:06 (1758823866) [ 254.708946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.031329] LustreError: 18852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.085273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.834944] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 260.578795] LustreError: 18852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.833332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 264.145803] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 267.428661] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 269.742899] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 272.525448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 274.675199] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 276.621751] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 295.391932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 295.395364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.401807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 295.404613] Lustre: Skipped 6 previous similar messages [ 299.564106] LustreError: 21120:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 299.566759] LustreError: 21120:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 299.628957] Lustre: server umount lustre-OST0000 complete [ 303.072204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 303.075819] LustreError: Skipped 1 previous similar message [ 315.359126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.419862] Lustre: server umount lustre-MDT0000 complete [ 316.971359] LustreError: 18831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823933 with bad export cookie 10428378005783889648 [ 316.974868] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.976196] LustreError: 18831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.045924] LustreError: 21526:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 317.048854] LustreError: 21526:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 317.155161] Lustre: server umount lustre-MDT0001 complete [ 321.708662] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 322.933097] Key type lgssc unregistered [ 323.094516] LNet: 22058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.097165] LNetError: 22058:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 323.107335] LNet: Removed LNI 192.168.204.121@tcp [ 323.502149] Key type .llcrypt unregistered [ 323.503306] Key type ._llcrypt unregistered [ 333.383344] Key type ._llcrypt registered [ 333.384483] Key type .llcrypt registered [ 333.425742] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 333.904131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 333.912479] alg: No test for adler32 (adler32-zlib) [ 334.775128] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 334.868536] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 336.455180] Key type lgssc registered [ 336.856846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 339.698417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 339.713489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.941751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 342.563671] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 345.892471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 346.021072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 347.371021] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 349.298345] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 350.682435] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 353.160379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 353.296644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.544841] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 357.726824] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 365.602821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 370.656905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.657532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 370.662903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 370.667704] Lustre: Skipped 1 previous similar message [ 373.612874] LustreError: 26129:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 373.681373] Lustre: server umount lustre-OST0000 complete [ 375.776759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.777475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.782046] Lustre: Skipped 2 previous similar messages [ 380.896331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 380.898713] Lustre: Skipped 2 previous similar messages [ 381.186438] LustreError: 26330:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 381.188751] LustreError: 26330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 381.210901] Lustre: server umount lustre-MDT0000 complete [ 382.549387] LustreError: 23419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758823999 with bad export cookie 8816701067794797761 [ 382.552183] LustreError: 23419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 382.552901] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.613586] LustreError: 26532:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 382.615918] LustreError: 26532:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 382.689169] Lustre: server umount lustre-MDT0001 complete [ 386.304514] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 387.355314] Key type lgssc unregistered [ 387.492367] LNet: 27062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.495516] LNetError: 27062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 387.506312] LNet: Removed LNI 192.168.204.121@tcp [ 387.812930] Key type .llcrypt unregistered [ 387.814187] Key type ._llcrypt unregistered [ 396.652791] Key type ._llcrypt registered [ 396.654383] Key type .llcrypt registered [ 396.699728] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 397.128699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 397.143110] alg: No test for adler32 (adler32-zlib) [ 397.996955] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 398.080388] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 399.671177] Key type lgssc registered [ 400.037059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 402.713104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 402.721736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.896224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.379824] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 408.466667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.627722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.179466] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 412.330899] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 413.732058] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 419.468772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 419.621087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.874314] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 423.830489] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 429.860939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 439.266241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.267571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.270167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.275269] Lustre: Skipped 1 previous similar message [ 442.987075] LustreError: 31045:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 443.046326] Lustre: server umount lustre-OST0000 complete [ 449.504618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.504754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 449.505529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.508706] Lustre: Skipped 2 previous similar messages [ 450.621514] LustreError: 31247:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 450.623124] LustreError: 31247:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 450.649825] Lustre: server umount lustre-MDT0000 complete [ 452.075140] LustreError: 28414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824068 with bad export cookie 9839936609861647314 [ 452.078139] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.079234] LustreError: 28414:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.154837] LustreError: 31449:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 452.158086] LustreError: 31449:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 452.252588] Lustre: server umount lustre-MDT0001 complete [ 456.715397] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 457.904212] Key type lgssc unregistered [ 458.049501] LNet: 31980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 458.053159] LNetError: 31980:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 458.060516] LNet: Removed LNI 192.168.204.121@tcp [ 458.376057] Key type .llcrypt unregistered [ 458.377728] Key type ._llcrypt unregistered [ 468.030261] Key type ._llcrypt registered [ 468.031588] Key type .llcrypt registered [ 468.074769] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 468.529894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 468.574532] alg: No test for adler32 (adler32-zlib) [ 469.457405] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 469.560079] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 471.151225] Key type lgssc registered [ 471.579462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 474.596585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 474.607212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.803285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.350209] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 480.405823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 480.545825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.031270] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 484.126308] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 485.485044] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 488.149146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 488.294947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.563869] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 492.866220] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 500.582109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 510.949303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.949590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.950584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 510.952573] LustreError: Skipped 1 previous similar message [ 510.962777] Lustre: Skipped 1 previous similar message [ 512.622444] LustreError: 35616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 512.701769] Lustre: server umount lustre-OST0000 complete [ 516.065134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.071994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.074379] Lustre: Skipped 2 previous similar messages [ 520.331096] LustreError: 35819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 520.334173] LustreError: 35819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 520.384789] Lustre: server umount lustre-MDT0000 complete [ 521.184986] LustreError: 33352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.193541] LustreError: 33352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 522.063582] LustreError: 35618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824138 with bad export cookie 7904868940495605574 [ 522.068042] LustreError: 35618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 522.068169] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.135577] LustreError: 36020:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 522.138118] LustreError: 36020:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 522.233520] Lustre: server umount lustre-MDT0001 complete [ 526.359601] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 527.604807] Key type lgssc unregistered [ 527.754902] LNet: 36551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.758970] LNetError: 36551:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.768436] LNet: Removed LNI 192.168.204.121@tcp [ 528.136490] Key type .llcrypt unregistered [ 528.138259] Key type ._llcrypt unregistered [ 537.384942] Key type ._llcrypt registered [ 537.385949] Key type .llcrypt registered [ 537.423247] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 537.757758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 537.777064] alg: No test for adler32 (adler32-zlib) [ 538.636040] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 538.733826] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 540.311203] Key type lgssc registered [ 540.705029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 543.529298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 543.540430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.739615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.055966] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 549.002375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 549.128548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 550.479934] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 552.319312] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 553.585532] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 555.823285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 555.932758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.812814] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 559.631072] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 571.301485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 579.551910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 579.555647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.559526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 580.430797] Lustre: lustre-OST0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 581.600593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 581.602724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.602984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.611600] Lustre: Skipped 1 previous similar message [ 583.726091] LustreError: 40190:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 583.788195] Lustre: server umount lustre-OST0000 complete [ 596.880485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.881577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 596.885182] Lustre: Skipped 1 previous similar message [ 602.080051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 602.082443] Lustre: Skipped 2 previous similar messages [ 611.295184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 611.326847] LustreError: 40566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 611.328253] LustreError: 40566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 611.357331] Lustre: server umount lustre-MDT0000 complete [ 612.320073] LustreError: 37939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.325526] LustreError: 37939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 612.574996] LustreError: 37902:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824229 with bad export cookie 3659463541971692692 [ 612.577940] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.578559] LustreError: 37902:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 618.681068] LustreError: 40769:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 618.683080] LustreError: 40769:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 618.759812] Lustre: server umount lustre-MDT0001 complete [ 622.249243] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 623.312348] Key type lgssc unregistered [ 623.438427] LNet: 41300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.440526] LNetError: 41300:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.452325] LNet: Removed LNI 192.168.204.121@tcp [ 623.755923] Key type .llcrypt unregistered [ 623.757335] Key type ._llcrypt unregistered [ 632.642710] Key type ._llcrypt registered [ 632.644515] Key type .llcrypt registered [ 632.688200] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 633.048221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.104229] alg: No test for adler32 (adler32-zlib) [ 633.974577] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 634.071070] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 635.655173] Key type lgssc registered [ 636.075083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.152290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 639.162436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.340314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.731177] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 644.650317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 644.790051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 646.211236] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 648.141747] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 649.501325] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 651.984777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 652.109606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.305822] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 656.313600] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 665.446804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 670.691868] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.692862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 670.695422] Lustre: Skipped 1 previous similar message [ 670.700132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 672.077133] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.21@tcp (stopping) [ 672.079813] Lustre: Skipped 2 previous similar messages [ 675.518293] LustreError: 44942:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 675.538923] Lustre: server umount lustre-MDT0000 complete [ 675.808681] LustreError: 42671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.813686] LustreError: 42671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 676.881570] LustreError: 42652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824293 with bad export cookie 11617214623149650107 [ 676.885850] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.887643] LustreError: 42652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 677.190899] LustreError: 42667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.200137] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.21@tcp (stopping) [ 680.928202] LustreError: 42666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.928509] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 680.936918] Lustre: Skipped 1 previous similar message [ 680.938791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 687.072262] LustreError: 42666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.072595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 687.076895] LustreError: 42666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 687.079522] Lustre: Skipped 2 previous similar messages [ 691.167121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 691.204905] LustreError: 45144:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 691.207863] LustreError: 45144:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 691.286715] Lustre: server umount lustre-MDT0001 complete [ 703.152626] LustreError: 45368:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 703.154822] LustreError: 45368:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 703.226288] Lustre: server umount lustre-OST0000 complete [ 709.151424] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 710.402356] Key type lgssc unregistered [ 710.535451] LNet: 46204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.538878] LNetError: 46204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.548625] LNet: Removed LNI 192.168.204.121@tcp [ 710.880281] Key type .llcrypt unregistered [ 710.882071] Key type ._llcrypt unregistered [ 720.513411] Key type ._llcrypt registered [ 720.514794] Key type .llcrypt registered [ 720.552331] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 720.974586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.983750] alg: No test for adler32 (adler32-zlib) [ 721.832495] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 721.918874] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 723.503167] Key type lgssc registered [ 723.871811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 726.491351] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 726.499461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 741.919310] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 747.039261] LustreError: 47546:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 773.663196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.632601] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 777.525839] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.321411] LustreError: 48254:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 943.387303] Lustre: server umount lustre-OST0000 complete [ 949.179862] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 950.463279] Key type lgssc unregistered [ 950.611716] LNet: 49090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.615192] LNetError: 49090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.624579] LNet: Removed LNI 192.168.204.121@tcp [ 950.947113] Key type .llcrypt unregistered [ 950.948787] Key type ._llcrypt unregistered [ 960.425063] Key type ._llcrypt registered [ 960.426509] Key type .llcrypt registered [ 960.463848] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 960.918637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.989651] alg: No test for adler32 (adler32-zlib) [ 961.865061] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 961.945362] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 963.527255] Key type lgssc registered [ 963.910354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 966.675517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 966.684328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.837205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.269309] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 972.242306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 972.376748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 973.791681] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 975.704713] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 977.006494] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 979.508279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 979.646386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.611385] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 983.592441] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 988.127966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.131312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.137940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.512172] LustreError: 52707:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 990.588775] Lustre: server umount lustre-OST0000 complete [ 992.061096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.061969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.065726] Lustre: Skipped 1 previous similar message [ 997.344939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.347491] Lustre: Skipped 2 previous similar messages [ 998.203239] LustreError: 52908:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 998.205181] LustreError: 52908:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 998.231756] Lustre: server umount lustre-MDT0000 complete [ 999.580524] LustreError: 51184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824616 with bad export cookie 15029440732337341535 [ 999.584229] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 999.585882] LustreError: 51184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 999.652583] LustreError: 53109:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 999.654350] LustreError: 53109:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 999.732424] Lustre: server umount lustre-MDT0001 complete [ 1003.443824] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1004.544408] Key type lgssc unregistered [ 1004.677387] LNet: 53642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.680150] LNetError: 53642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.689323] LNet: Removed LNI 192.168.204.121@tcp [ 1004.994049] Key type .llcrypt unregistered [ 1004.995409] Key type ._llcrypt unregistered [ 1013.999843] Key type ._llcrypt registered [ 1014.001157] Key type .llcrypt registered [ 1014.043278] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1014.402605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.443094] alg: No test for adler32 (adler32-zlib) [ 1015.316680] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1015.414826] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1016.999191] Key type lgssc registered [ 1017.404210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.342930] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.351343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1035.807276] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1040.927205] LustreError: 54984:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 1067.551202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.462103] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1071.273061] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1073.526144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.693766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.719358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1074.999920] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1077.872358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1078.017163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.410500] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1081.270794] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.530614] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1083.360167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.363220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.368681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.871893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.875868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.881830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1088.483360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1088.485290] Lustre: Skipped 1 previous similar message [ 1089.261048] LustreError: 57260:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1089.322370] Lustre: server umount lustre-OST0000 complete [ 1103.350309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.350791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.354223] Lustre: Skipped 1 previous similar message [ 1103.357133] Lustre: Skipped 1 previous similar message [ 1108.448065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.450104] Lustre: Skipped 1 previous similar message [ 1109.562939] LustreError: 57637:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1109.564722] LustreError: 57637:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1109.587846] Lustre: server umount lustre-MDT0000 complete [ 1110.757682] LustreError: 54991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824727 with bad export cookie 9582620257136016945 [ 1110.760912] LustreError: 54991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1110.761454] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.343119] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1125.376841] LustreError: 57838:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1125.379034] LustreError: 57838:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1125.466923] Lustre: server umount lustre-MDT0001 complete [ 1128.824177] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1129.859682] Key type lgssc unregistered [ 1129.987433] LNet: 58371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.990148] LNetError: 58371:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.999334] LNet: Removed LNI 192.168.204.121@tcp [ 1130.308283] Key type .llcrypt unregistered [ 1130.309704] Key type ._llcrypt unregistered [ 1138.488287] Key type ._llcrypt registered [ 1138.489437] Key type .llcrypt registered [ 1138.522849] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1138.796531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1138.842124] alg: No test for adler32 (adler32-zlib) [ 1139.683985] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1139.759628] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1141.335132] Key type lgssc registered [ 1141.673550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.081586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.088410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.253244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.603098] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1149.599887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1149.713128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1151.003915] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1152.619699] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1153.824100] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.024167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1156.136715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.870781] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1159.473764] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1169.446404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1174.497404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1174.498168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1174.500183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.503938] Lustre: Skipped 1 previous similar message [ 1174.506524] Lustre: Skipped 1 previous similar message [ 1178.093369] LustreError: 62007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1178.157544] Lustre: server umount lustre-OST0000 complete [ 1179.616369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.620700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.622818] Lustre: Skipped 1 previous similar message [ 1184.736706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1184.739372] Lustre: Skipped 1 previous similar message [ 1185.467237] LustreError: 62209:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1185.468920] LustreError: 62209:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1185.495420] Lustre: server umount lustre-MDT0000 complete [ 1186.672927] LustreError: 59720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824803 with bad export cookie 6145738470078596594 [ 1186.675456] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.676556] LustreError: 59720:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1186.739204] LustreError: 62410:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1186.741187] LustreError: 62410:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1186.813957] Lustre: server umount lustre-MDT0001 complete [ 1190.055270] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1191.038437] Key type lgssc unregistered [ 1191.160350] LNet: 62941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.162723] LNetError: 62941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.171471] LNet: Removed LNI 192.168.204.121@tcp [ 1191.444872] Key type .llcrypt unregistered [ 1191.445835] Key type ._llcrypt unregistered [ 1202.622370] Key type ._llcrypt registered [ 1202.623320] Key type .llcrypt registered [ 1202.662409] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1202.991763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.039961] alg: No test for adler32 (adler32-zlib) [ 1203.883223] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1203.964497] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1205.535150] Key type lgssc registered [ 1205.948184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.621151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.628504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.799475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.124200] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1214.136840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1214.271343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1215.756581] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1217.710167] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1218.991725] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1221.163392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1221.264606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.095222] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1224.849952] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1234.598330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1237.433548] Lustre: Failing over lustre-MDT0000 [ 1237.551386] LustreError: 66867:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1237.576944] Lustre: server umount lustre-MDT0000 complete [ 1240.034853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1240.036969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.041081] LustreError: 64602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.043745] Lustre: Skipped 2 previous similar messages [ 1240.049718] LustreError: 64602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1240.942787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.988090] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.050602] Lustre: *** cfs_fail_loc=135, val=0*** [ 1241.092053] LustreError: 67378:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1241.094369] LustreError: 67378:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -2 [ 1241.097603] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1241.097603] [ 1241.101209] LustreError: MGC192.168.204.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1241.107892] LustreError: 67366:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1241.111687] LustreError: 67366:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1241.115935] LustreError: 67366:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1241.127342] Lustre: server umount lustre-MDT0000 complete [ 1241.128903] LustreError: 67366:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1243.525789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.561818] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.650684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.682329] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1245.051586] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1245.510237] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1248.743510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1248.753573] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1248.770511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1253.858923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1253.859046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.861556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1253.864414] Lustre: Skipped 1 previous similar message [ 1253.867021] Lustre: Skipped 1 previous similar message [ 1255.785411] LustreError: 68179:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1255.787443] LustreError: 68179:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1255.840725] Lustre: server umount lustre-OST0000 complete [ 1258.977932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.978635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1258.979077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.981407] Lustre: Skipped 1 previous similar message [ 1264.096687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.099396] Lustre: Skipped 2 previous similar messages [ 1269.216775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.218729] Lustre: Skipped 1 previous similar message [ 1271.263205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1271.296460] LustreError: 68384:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1271.298315] LustreError: 68384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1271.326182] Lustre: server umount lustre-MDT0000 complete [ 1272.661339] LustreError: 64584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824889 with bad export cookie 4302316750031669520 [ 1272.664809] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.666600] LustreError: 64584:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1272.830587] Lustre: server umount lustre-MDT0001 complete [ 1276.652080] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1277.752643] Key type lgssc unregistered [ 1277.895541] LNet: 69117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.899558] LNetError: 69117:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.905389] LNet: Removed LNI 192.168.204.121@tcp [ 1278.198412] Key type .llcrypt unregistered [ 1278.199786] Key type ._llcrypt unregistered [ 1286.800026] Key type ._llcrypt registered [ 1286.801311] Key type .llcrypt registered [ 1286.840986] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1287.142705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.159067] alg: No test for adler32 (adler32-zlib) [ 1288.003759] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1288.088980] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1289.671157] Key type lgssc registered [ 1290.036383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.701621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1292.708781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.869632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.210496] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1297.899549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1298.003516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.165284] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1300.975779] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1302.255397] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1304.659404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1304.766798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.583447] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1308.373688] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1319.142743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1326.063795] Lustre: Failing over lustre-MDT0000 [ 1326.295647] LustreError: 72756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1326.328452] Lustre: server umount lustre-MDT0000 complete [ 1328.607935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1328.611133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.617946] LustreError: 71203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.627747] LustreError: 71203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1329.601879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.632387] LustreError: 71203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.633854] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.643816] LustreError: 71203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1329.659228] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1329.721701] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1329.777089] LustreError: 73266:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1329.779840] LustreError: 73266:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -9 [ 1329.783218] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1329.783218] [ 1329.787073] LustreError: MGC192.168.204.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1329.793581] LustreError: 73254:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1329.796799] LustreError: 73254:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1329.801074] LustreError: 73254:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1329.812757] Lustre: server umount lustre-MDT0000 complete [ 1329.814204] LustreError: 73254:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1332.187812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.223368] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.303295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.332727] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1333.605633] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1334.085975] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1337.316952] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1337.323436] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1337.339275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1338.847880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1338.850192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1338.854103] Lustre: Skipped 1 previous similar message [ 1338.856335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1342.433581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1342.434767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1342.435139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.444305] Lustre: Skipped 1 previous similar message [ 1344.301962] LustreError: 74076:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1344.304464] LustreError: 74076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1344.369170] Lustre: server umount lustre-OST0000 complete [ 1347.552539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1347.552982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.559847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.562442] Lustre: Skipped 1 previous similar message [ 1351.674389] LustreError: 74277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1351.676070] LustreError: 74277:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1351.704744] Lustre: server umount lustre-MDT0000 complete [ 1352.671849] LustreError: 70490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.677446] LustreError: 70490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1352.830032] LustreError: 72107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758824969 with bad export cookie 8963387509488271746 [ 1352.831832] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.833888] LustreError: 72107:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1352.959216] Lustre: server umount lustre-MDT0001 complete [ 1356.414342] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1357.409358] Key type lgssc unregistered [ 1357.537378] LNet: 75009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.540635] LNetError: 75009:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.547303] LNet: Removed LNI 192.168.204.121@tcp [ 1357.842362] Key type .llcrypt unregistered [ 1357.843248] Key type ._llcrypt unregistered [ 1361.572276] Key type ._llcrypt registered [ 1361.573172] Key type .llcrypt registered [ 1361.608587] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1361.711503] Key type .llcrypt unregistered [ 1361.713242] Key type ._llcrypt unregistered [ 1370.413218] Key type ._llcrypt registered [ 1370.414396] Key type .llcrypt registered [ 1370.454288] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1370.913029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1370.924439] alg: No test for adler32 (adler32-zlib) [ 1371.788574] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1371.876810] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1373.463171] Key type lgssc registered [ 1373.860120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1376.626715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1376.634181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1377.795179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.074323] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1381.917811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1382.033631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1383.290104] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1385.054547] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1386.240262] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1388.425815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1388.535089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.313211] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1391.591122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1391.949569] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1396.704932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1396.705664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.706699] LustreError: Skipped 1 previous similar message [ 1396.709969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1396.713544] Lustre: Skipped 1 previous similar message [ 1401.583107] LustreError: 79092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1401.646185] Lustre: server umount lustre-OST0000 complete [ 1406.944749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.945299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1406.948643] Lustre: Skipped 2 previous similar messages [ 1406.952236] Lustre: Skipped 1 previous similar message [ 1409.087154] LustreError: 79294:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1409.090805] LustreError: 79294:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1409.119062] Lustre: server umount lustre-MDT0000 complete [ 1410.453305] LustreError: 76827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825027 with bad export cookie 11168717212862869552 [ 1410.455847] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1410.458197] LustreError: 76827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1410.527831] LustreError: 79496:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1410.529956] LustreError: 79496:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1410.615199] Lustre: server umount lustre-MDT0001 complete [ 1414.388198] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1415.487709] Key type lgssc unregistered [ 1415.633423] LNet: 80027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.635561] LNetError: 80027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.643326] LNet: Removed LNI 192.168.204.121@tcp [ 1415.962533] Key type .llcrypt unregistered [ 1415.963495] Key type ._llcrypt unregistered [ 1424.815926] Key type ._llcrypt registered [ 1424.817905] Key type .llcrypt registered [ 1424.859298] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1425.190053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.215048] alg: No test for adler32 (adler32-zlib) [ 1426.071192] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1426.165686] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1427.751165] Key type lgssc registered [ 1428.152025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1430.945272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1430.952191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1432.114148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.531834] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1436.464312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1436.593936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1438.064368] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1439.944705] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1441.181726] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1443.510773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1443.618026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.530134] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1447.271742] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1456.936295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1462.242906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.243251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1462.247545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1462.251447] Lustre: Skipped 1 previous similar message [ 1466.349098] LustreError: 83665:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1466.421616] Lustre: server umount lustre-OST0000 complete [ 1472.480543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1472.480703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.482338] LustreError: Skipped 1 previous similar message [ 1472.487409] Lustre: Skipped 1 previous similar message [ 1472.489707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.491874] Lustre: Skipped 1 previous similar message [ 1473.722432] LustreError: 83866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1473.724335] LustreError: 83866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1473.747590] Lustre: server umount lustre-MDT0000 complete [ 1474.967179] LustreError: 82116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825091 with bad export cookie 16859802527013710768 [ 1474.969902] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.971473] LustreError: 82116:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1475.031112] LustreError: 84068:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1475.032453] LustreError: 84068:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1475.127678] Lustre: server umount lustre-MDT0001 complete [ 1478.574480] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1479.571393] Key type lgssc unregistered [ 1479.700329] LNet: 84599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.702585] LNetError: 84599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.714310] LNet: Removed LNI 192.168.204.121@tcp [ 1479.994159] Key type .llcrypt unregistered [ 1479.995829] Key type ._llcrypt unregistered [ 1488.583493] Key type ._llcrypt registered [ 1488.585210] Key type .llcrypt registered [ 1488.626757] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1488.940737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1488.973022] alg: No test for adler32 (adler32-zlib) [ 1489.828960] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1489.920145] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1491.503113] Key type lgssc registered [ 1491.866563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1494.333075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1494.339198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1495.488246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.740548] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1499.620848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1499.733640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1501.026114] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1502.769949] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1503.931481] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1506.178529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1506.297647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1508.138043] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1509.923402] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1520.613643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1529.823994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1529.827812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1529.834076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1530.849996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1530.851577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.851921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.832169] LustreError: 88242:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1534.900821] Lustre: server umount lustre-OST0000 complete [ 1540.575900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1540.579307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1540.585443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1540.587656] Lustre: Skipped 1 previous similar message [ 1542.334601] LustreError: 88444:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1542.336584] LustreError: 88444:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1542.362558] Lustre: server umount lustre-MDT0000 complete [ 1543.711771] LustreError: 86690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825160 with bad export cookie 9672560265538960494 [ 1543.715653] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.717115] LustreError: 86690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1543.787245] LustreError: 88645:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1543.789750] LustreError: 88645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1543.870451] Lustre: server umount lustre-MDT0001 complete [ 1547.860683] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1549.127326] Key type lgssc unregistered [ 1549.281571] LNet: 89176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.285610] LNetError: 89176:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.293396] LNet: Removed LNI 192.168.204.121@tcp [ 1549.692073] Key type .llcrypt unregistered [ 1549.693579] Key type ._llcrypt unregistered [ 1560.280106] Key type ._llcrypt registered [ 1560.281844] Key type .llcrypt registered [ 1560.338926] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1560.832238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1560.843800] alg: No test for adler32 (adler32-zlib) [ 1561.762479] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1561.906940] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1563.551206] Key type lgssc registered [ 1564.163499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1568.037519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1568.046915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1583.519300] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1588.639208] LustreError: 90518:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 1615.263170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.148800] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1618.960200] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1621.056515] LustreError: 91377:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1621.117836] Lustre: server umount lustre-OST0000 complete [ 1624.000798] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:34:00 (1758825240) [ 1626.379369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1626.553391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.053245] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1631.037174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1631.166045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1632.523333] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1634.370536] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1635.691856] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1638.135337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1638.252346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.140727] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1641.921414] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1651.558952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1656.804176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.804959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1656.805518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.809770] Lustre: Skipped 1 previous similar message [ 1658.029094] LustreError: 94472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1658.030862] LustreError: 94472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1658.095738] Lustre: server umount lustre-OST0000 complete [ 1661.499613] LustreError: 94673:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1661.501600] LustreError: 94673:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1661.525899] Lustre: server umount lustre-MDT0000 complete [ 1661.920920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.922597] LustreError: 92046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.925193] Lustre: Skipped 1 previous similar message [ 1661.930302] LustreError: 92046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1661.932073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.854639] LustreError: 92029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825279 with bad export cookie 1186772922583524156 [ 1662.858155] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1662.858256] LustreError: 92029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1663.018468] Lustre: server umount lustre-MDT0001 complete [ 1666.754495] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1667.919156] Key type lgssc unregistered [ 1668.048588] LNet: 95405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.050956] LNetError: 95405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.060326] LNet: Removed LNI 192.168.204.121@tcp [ 1668.359968] Key type .llcrypt unregistered [ 1668.361228] Key type ._llcrypt unregistered [ 1677.137207] Key type ._llcrypt registered [ 1677.138632] Key type .llcrypt registered [ 1677.177343] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1677.548631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.575824] alg: No test for adler32 (adler32-zlib) [ 1678.455869] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1678.550264] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1680.135139] Key type lgssc registered [ 1680.483748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1683.016698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1683.023478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.165536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.536378] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1688.449428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1688.574529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1689.948706] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1691.725156] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1692.949455] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1695.186398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.293890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.154239] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1698.963372] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1709.666569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1719.264334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.267703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1719.274197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.289269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1720.290325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1720.291109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.301941] Lustre: Skipped 1 previous similar message [ 1723.182099] LustreError: 99043:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1723.252699] Lustre: server umount lustre-OST0000 complete [ 1725.408716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.409418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.412513] Lustre: Skipped 1 previous similar message [ 1725.417508] Lustre: Skipped 1 previous similar message [ 1730.528575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1730.531250] Lustre: Skipped 1 previous similar message [ 1730.685064] LustreError: 99244:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1730.687628] LustreError: 99244:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1730.713977] Lustre: server umount lustre-MDT0000 complete [ 1732.081256] LustreError: 99246:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825348 with bad export cookie 9425343833392767476 [ 1732.084480] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.086316] LustreError: 99246:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1732.152603] LustreError: 99447:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1732.154868] LustreError: 99447:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1732.232737] Lustre: server umount lustre-MDT0001 complete [ 1736.169793] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 1737.304946] Key type lgssc unregistered [ 1737.452479] LNet: 99979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.455228] LNetError: 99979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.461356] LNet: Removed LNI 192.168.204.121@tcp [ 1737.773656] Key type .llcrypt unregistered [ 1737.775513] Key type ._llcrypt unregistered [ 1745.714653] Key type ._llcrypt registered [ 1745.715694] Key type .llcrypt registered [ 1745.760177] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1746.235139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.247591] alg: No test for adler32 (adler32-zlib) [ 1747.136266] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 1747.236832] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 1748.839150] Key type lgssc registered [ 1749.247484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1752.306233] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1752.314812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.775319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1772.895269] LustreError: 101179:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 1799.519168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.480760] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1803.281589] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1805.635820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.748953] LustreError: MGC192.168.204.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1805.754381] LustreError: 102089:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1805.757659] LustreError: 102089:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1805.761746] LustreError: 102089:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 1805.821988] Lustre: server umount lustre-MDT0000 complete [ 1805.824071] LustreError: 102089:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1819.103590] LustreError: 102521:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1819.182633] Lustre: server umount lustre-OST0000 complete [ 1822.296433] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1825.513307] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1828.914747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1831.151835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1833.137047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1835.097127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1837.323500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1837.352421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.454869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1837.468691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1837.510497] Lustre: lustre-MDT0000: new disk, initializing [ 1837.536961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.543891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1838.818244] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1842.922580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1842.945447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1842.975848] Lustre: 104853:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.999070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1843.001648] Lustre: Skipped 1 previous similar message [ 1843.038606] Lustre: lustre-MDT0001: new disk, initializing [ 1843.075257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.086549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1843.091830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1844.317727] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1847.428429] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1848.653494] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1851.007563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1851.033226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1851.130728] Lustre: lustre-OST0000: new disk, initializing [ 1851.132557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1851.162249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.358375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1851.363643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1851.397724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1853.165205] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1856.341443] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1857.775668] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1857.861060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1859.275744] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1859.359548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1861.601080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.602184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.604347] LustreError: Skipped 1 previous similar message [ 1861.608141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1861.611913] Lustre: Skipped 1 previous similar message [ 1866.722037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.723948] Lustre: Skipped 1 previous similar message [ 1871.842222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1871.844041] Lustre: Skipped 1 previous similar message [ 1874.399124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1874.418126] LustreError: 107795:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1874.421325] LustreError: 107795:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1874.483054] Lustre: server umount lustre-OST0000 complete [ 1876.960626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.961303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.966692] Lustre: Skipped 2 previous similar messages [ 1882.080277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.082168] Lustre: Skipped 2 previous similar messages [ 1890.271118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1890.305785] LustreError: 107997:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1890.308699] LustreError: 107997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1890.339337] Lustre: server umount lustre-MDT0000 complete [ 1891.521281] LustreError: 106603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825508 with bad export cookie 10228372828725602754 [ 1891.523847] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.524737] LustreError: 106603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1891.656180] Lustre: server umount lustre-MDT0001 complete [ 1894.230427] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:38:30 (1758825510) [ 1895.260549] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1900.210780] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 1902.893835] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 1906.185544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1908.443445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1910.274883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1912.221170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1914.518111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1914.546266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.626893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1914.637561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1914.674348] Lustre: lustre-MDT0000: new disk, initializing [ 1914.699847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.706970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1916.079540] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1920.349114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1920.380748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1920.422336] Lustre: 111328:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1920.447290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1920.449270] Lustre: Skipped 1 previous similar message [ 1920.491922] Lustre: lustre-MDT0001: new disk, initializing [ 1920.539079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1920.542607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1921.953589] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1925.119394] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1926.374104] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1928.701458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1928.730221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1928.833727] Lustre: lustre-OST0000: new disk, initializing [ 1928.836944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1928.868741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.871266] Lustre: Skipped 1 previous similar message [ 1930.948890] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1934.243415] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1934.326609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1934.330027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1934.360037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1935.540693] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1935.614241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1936.944879] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1937.020271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1939.424442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.425106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.427291] LustreError: Skipped 1 previous similar message [ 1939.431651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.434186] Lustre: Skipped 3 previous similar messages [ 1943.725583] LustreError: 114265:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1943.727382] LustreError: 114265:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1943.792926] Lustre: server umount lustre-OST0000 complete [ 1946.079891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1946.083795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.089918] Lustre: Skipped 1 previous similar message [ 1959.391123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1959.422680] LustreError: 114467:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1959.425106] LustreError: 114467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1959.450910] Lustre: server umount lustre-MDT0000 complete [ 1960.672106] LustreError: 111319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825577 with bad export cookie 10228372828725603622 [ 1960.674802] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.676342] LustreError: 111319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1960.814683] Lustre: server umount lustre-MDT0001 complete [ 1963.671431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.856484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.197405] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1968.011018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1969.400802] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1971.101762] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1972.207392] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1974.301310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1976.233953] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 1977.915390] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1996.048531] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1998.303984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1998.306747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.311358] Lustre: Skipped 1 previous similar message [ 1998.313844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.316037] Lustre: Skipped 5 previous similar messages [ 2003.116822] LustreError: 117510:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2003.118755] LustreError: 117510:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2003.180606] Lustre: server umount lustre-OST0000 complete [ 2010.577658] Lustre: server umount lustre-MDT0000 complete [ 2011.865402] LustreError: 115126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825628 with bad export cookie 10228372828725604490 [ 2011.868687] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.869023] LustreError: 115126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2012.014500] Lustre: server umount lustre-MDT0001 complete [ 2015.688286] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2016.816837] Key type lgssc unregistered [ 2016.954445] LNet: 118447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.957301] LNetError: 118447:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.966340] LNet: Removed LNI 192.168.204.121@tcp [ 2017.311550] Key type .llcrypt unregistered [ 2017.313217] Key type ._llcrypt unregistered [ 2023.395980] Key type ._llcrypt registered [ 2023.397589] Key type .llcrypt registered [ 2023.432581] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 2029.543737] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2029.980519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.990057] alg: No test for adler32 (adler32-zlib) [ 2030.855190] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2030.937505] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2032.519170] Key type lgssc registered [ 2032.944420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2036.117830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2038.382296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2040.425227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2042.421884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2044.757612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2044.783118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2044.789584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.878659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2045.889427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2045.931584] Lustre: lustre-MDT0000: new disk, initializing [ 2045.956918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.964622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2047.294759] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2051.552323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2051.573220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2051.610442] Lustre: 121547:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.637878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2051.640565] Lustre: Skipped 1 previous similar message [ 2051.681748] Lustre: lustre-MDT0001: new disk, initializing [ 2051.716204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.727722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2051.731227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2053.091627] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2056.251814] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2057.492150] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2059.639750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2059.663047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2059.762858] Lustre: lustre-OST0000: new disk, initializing [ 2059.764875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2059.796286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.631636] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2062.327815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2062.331718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2062.363026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2064.673107] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2065.997387] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2066.067078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2067.384537] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2067.460974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2072.546133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2072.547180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2072.548599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2072.552191] Lustre: Skipped 1 previous similar message [ 2072.555183] Lustre: Skipped 1 previous similar message [ 2074.157158] LustreError: 124482:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2074.220178] Lustre: server umount lustre-OST0000 complete [ 2077.665042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.665097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2077.670821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.671905] LustreError: Skipped 1 previous similar message [ 2077.675596] Lustre: Skipped 1 previous similar message [ 2081.664719] LustreError: 124683:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2081.666503] LustreError: 124683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2081.701835] Lustre: server umount lustre-MDT0000 complete [ 2082.965292] LustreError: 121538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825699 with bad export cookie 7208516350248776478 [ 2082.968333] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.968336] LustreError: 121538:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2083.030744] LustreError: 124885:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2083.033316] LustreError: 124885:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2083.123800] Lustre: server umount lustre-MDT0001 complete [ 2085.961786] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 14:41:42 (1758825702) [ 2088.196394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.344100] LustreError: 125555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2088.349983] LustreError: 125555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2088.367289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.695684] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2092.542858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2092.668929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2093.960114] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2095.769745] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2097.011267] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2097.683382] LustreError: 125583:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2097.685220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.685884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.688240] LustreError: 125583:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2097.693027] Lustre: Skipped 2 previous similar messages [ 2097.695526] LustreError: 125583:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2102.752206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.754787] Lustre: Skipped 2 previous similar messages [ 2103.801993] LustreError: 126894:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2103.803861] LustreError: 126894:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2103.828460] Lustre: server umount lustre-MDT0000 complete [ 2105.055037] LustreError: 125537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825721 with bad export cookie 7208516350248777353 [ 2105.058246] LustreError: 125537:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2105.058286] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.194560] Lustre: server umount lustre-MDT0001 complete [ 2107.941330] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 14:42:04 (1758825724) [ 2110.129882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2125.599549] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2130.719533] LustreError: 127740:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 2156.319302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.336709] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2160.296775] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2161.016483] LustreError: 128404:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2161.018495] LustreError: 128404:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2161.086505] Lustre: server umount lustre-OST0000 complete [ 2163.762612] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 14:42:59 (1758825779) [ 2165.987263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.153529] LustreError: 129074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.542811] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2170.417618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2171.870434] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2173.671067] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2174.813019] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2177.007777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2177.123075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.124968] Lustre: Skipped 2 previous similar messages [ 2178.998609] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2180.782921] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2195.940126] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2195.942138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2195.943492] Lustre: Skipped 2 previous similar messages [ 2195.947969] Lustre: Skipped 4 previous similar messages [ 2198.780072] LustreError: 131343:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2198.781858] LustreError: 131343:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2198.809556] Lustre: server umount lustre-MDT0000 complete [ 2200.075156] LustreError: 129055:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825816 with bad export cookie 7208516350248777983 [ 2200.079230] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.080230] LustreError: 129055:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2200.221825] Lustre: server umount lustre-MDT0001 complete [ 2211.759366] Lustre: server umount lustre-OST0000 complete [ 2214.601602] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 14:43:50 (1758825830) [ 2216.875628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.034355] LustreError: 132418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.056691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.343764] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2221.253021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2222.725540] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2224.531559] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2225.752043] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2228.069635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2230.102139] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2231.900908] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2233.268640] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2241.510309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2242.553335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2243.978731] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2244.077920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2246.626986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.627102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2246.627807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2246.632573] Lustre: Skipped 1 previous similar message [ 2250.796929] LustreError: 135224:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2250.799280] LustreError: 135224:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2250.858245] Lustre: server umount lustre-OST0000 complete [ 2252.255870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2256.864985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.869093] Lustre: Skipped 1 previous similar message [ 2258.194950] Lustre: server umount lustre-MDT0000 complete [ 2259.416248] LustreError: 132398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825876 with bad export cookie 7208516350248778998 [ 2259.419377] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.422402] LustreError: 132398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2259.566047] Lustre: server umount lustre-MDT0001 complete [ 2262.354412] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 14:44:38 (1758825878) [ 2264.622923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2280.031634] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2285.151447] LustreError: 136270:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 2310.751634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.753694] Lustre: Skipped 2 previous similar messages [ 2312.619434] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2314.310226] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2316.449681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.591252] LustreError: 136297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.641326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2317.858375] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2320.653251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2322.068351] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2323.795635] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2324.998048] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2326.352936] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2326.427681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2327.766578] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2327.847041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2331.109473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.109572] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.111409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.111414] Lustre: Skipped 4 previous similar messages [ 2331.118498] Lustre: Skipped 1 previous similar message [ 2334.591229] LustreError: 139043:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2334.594215] LustreError: 139043:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2334.620819] Lustre: server umount lustre-MDT0000 complete [ 2335.903239] LustreError: 136278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758825952 with bad export cookie 7208516350248780006 [ 2335.906061] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.906579] LustreError: 136278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2336.062175] Lustre: server umount lustre-MDT0001 complete [ 2346.483850] Lustre: server umount lustre-OST0000 complete [ 2349.322727] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 14:46:05 (1758825965) [ 2351.551167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2367.007805] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2372.127469] LustreError: 140090:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 2399.587223] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2401.336441] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2403.581526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.748107] LustreError: 140118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.804622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2405.157527] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2407.975899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2409.400062] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2411.116305] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2412.270872] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2414.614334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.640141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2414.696866] Lustre: lustre-OST0001: new disk, initializing [ 2414.698781] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2416.689629] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2419.782973] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2421.078581] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2421.237467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2421.240561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2421.271201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2422.169786] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2423.473743] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2423.546307] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2424.287803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.291457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.295933] Lustre: Skipped 1 previous similar message [ 2424.299043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.300829] Lustre: Skipped 2 previous similar messages [ 2430.252242] LustreError: 143829:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2430.255621] LustreError: 143829:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2430.269223] Lustre: server umount lustre-OST0000 complete [ 2431.464911] LustreError: 141040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.470681] LustreError: 141040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2437.679461] Lustre: server umount lustre-OST0001 complete [ 2445.016119] Lustre: server umount lustre-MDT0000 complete [ 2446.234116] LustreError: 140097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826062 with bad export cookie 7208516350248780951 [ 2446.237556] LustreError: 140097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2446.237642] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2446.378332] Lustre: server umount lustre-MDT0001 complete [ 2448.224940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2449.478670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.868850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2452.192552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2454.379218] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 14:47:50 (1758826070) [ 2454.914557] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2455.434299] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2455.969816] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 14:47:52 (1758826072) [ 2458.212850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.308280] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2458.319922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2458.395001] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.398223] Lustre: Skipped 6 previous similar messages [ 2459.722789] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2462.891641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2462.941583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2462.953680] Lustre: 146067:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2464.543922] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2466.668617] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2468.017589] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2470.426438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2470.494475] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2472.597646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2472.666868] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2474.696546] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2476.204066] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2476.287123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2477.628519] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2477.704795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2483.168512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2483.171591] LustreError: Skipped 4 previous similar messages [ 2484.516448] Lustre: server umount lustre-OST0000 complete [ 2497.973821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2499.954608] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2501.664215] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2503.119791] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2504.227407] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.231389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.231469] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2504.237935] Lustre: Skipped 1 previous similar message [ 2505.231407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2506.388196] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2506.463153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2507.500299] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d37c8cdd000.ost_server_uuid 50 [ 2507.967978] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d37c8cdd000.ost_server_uuid in FULL state after 0 sec [ 2508.021540] LustreError: 147207:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.204.21@tcp) returned error from blocking AST (req@00000000c399e41a x1844262361811328 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000037bd9136/0x6409cb6d9e2c8ddc lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.21@tcp remote: 0x18f026bca7437a55 expref: 5 pid: 146073 timeout: 2608 lvb_type: 0 [ 2508.032078] LustreError: lustre-MDT0000: A client on nid 192.168.204.21@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2508.034634] LustreError: 146062:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.21@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000037bd9136/0x6409cb6d9e2c8ddc lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.21@tcp remote: 0x18f026bca7437a55 expref: 6 pid: 146073 timeout: 0 lvb_type: 0 [ 2509.282653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.283258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.285261] Lustre: Skipped 8 previous similar messages [ 2509.287704] Lustre: Skipped 9 previous similar messages [ 2515.179552] Lustre: server umount lustre-OST0000 complete [ 2519.521047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2519.523901] LustreError: Skipped 2 previous similar messages [ 2522.581024] Lustre: server umount lustre-MDT0000 complete [ 2523.811158] LustreError: 146794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826140 with bad export cookie 7208516350248782127 [ 2523.814761] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.817128] LustreError: 146794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2529.994427] Lustre: server umount lustre-MDT0001 complete [ 2533.577065] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2534.547342] Key type lgssc unregistered [ 2534.670348] LNet: 151567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.672588] LNetError: 151567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.682353] LNet: Removed LNI 192.168.204.121@tcp [ 2534.972641] Key type .llcrypt unregistered [ 2534.974218] Key type ._llcrypt unregistered [ 2542.967160] Key type ._llcrypt registered [ 2542.968121] Key type .llcrypt registered [ 2542.998821] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2543.282186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.298610] alg: No test for adler32 (adler32-zlib) [ 2544.148748] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2544.223979] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2545.799187] Key type lgssc registered [ 2546.158353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2548.635182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2548.641157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.786661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2550.934023] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2553.551056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2553.668643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.969993] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2556.763135] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2557.972348] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2560.174456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2560.278350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.069072] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2563.755114] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2574.629283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2581.286216] Lustre: Failing over lustre-MDT0000 [ 2581.482140] LustreError: 155228:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2581.511675] Lustre: server umount lustre-MDT0000 complete [ 2584.543966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2584.547256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.552816] LustreError: 152959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.556891] LustreError: 152959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2585.056689] LustreError: 153672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.057889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.061633] LustreError: 153672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2585.065518] Lustre: Skipped 1 previous similar message [ 2590.176235] LustreError: 153672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.180180] LustreError: 153672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2592.395504] LustreError: 152941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.21@tcp arrived at 1758826209 with bad export cookie 14883507211819315415 [ 2592.399817] LustreError: 152941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2594.736609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.777830] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.877507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.909907] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2596.196788] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2596.896190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2596.899831] Lustre: lustre-MDT0000: Denying connection for new client 5d1a691a-7dc9-4952-9b87-c25501932089 (at 192.168.204.21@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2599.908265] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2604.511877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.514373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.518621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.026447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.029484] Lustre: Skipped 1 previous similar message [ 2609.711110] LustreError: 156157:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2609.713151] LustreError: 156157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2609.786231] Lustre: server umount lustre-OST0000 complete [ 2611.024452] LustreError: 156361:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2611.028426] Lustre: 155728:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2611.032174] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2611.039227] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2611.044050] LustreError: 155728:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001f2ebe95 x1844262899583616/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2611.050050] LustreError: 155728:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2611.053311] LustreError: 155728:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2611.056492] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2611.130383] LustreError: 156361:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2611.132286] LustreError: 156361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2611.155963] Lustre: server umount lustre-MDT0000 complete [ 2612.388161] LustreError: 155353:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826229 with bad export cookie 14883507211819315772 [ 2612.392211] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.392644] LustreError: 152270:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2618.492216] LustreError: 156562:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2618.494149] LustreError: 156562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2618.572966] Lustre: server umount lustre-MDT0001 complete [ 2622.249959] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2623.316934] Key type lgssc unregistered [ 2623.452395] LNet: 157093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.455204] LNetError: 157093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.464416] LNet: Removed LNI 192.168.204.121@tcp [ 2623.801168] Key type .llcrypt unregistered [ 2623.802261] Key type ._llcrypt unregistered [ 2632.461708] Key type ._llcrypt registered [ 2632.462704] Key type .llcrypt registered [ 2632.501019] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2632.873863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2632.935355] alg: No test for adler32 (adler32-zlib) [ 2633.781397] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2633.859201] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2635.447211] Key type lgssc registered [ 2635.802746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2638.324770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2638.330877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.506580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.838915] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2643.716570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2643.839735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.126963] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2646.836503] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2648.016921] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2650.356729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2650.488772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2652.437938] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2654.308462] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2664.805570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2674.144369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2674.146213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.149640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.170026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.170406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.230062] LustreError: 160732:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2677.289105] Lustre: server umount lustre-OST0000 complete [ 2679.775992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2679.779912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.783950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2679.785606] Lustre: Skipped 1 previous similar message [ 2684.672286] LustreError: 160934:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2684.675173] LustreError: 160934:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2684.703260] Lustre: server umount lustre-MDT0000 complete [ 2685.408581] LustreError: 158483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.414182] LustreError: 158483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2685.976271] LustreError: 158445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826302 with bad export cookie 1314553155478290989 [ 2685.979586] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.981393] LustreError: 158445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2686.055246] LustreError: 161135:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2686.058593] LustreError: 161135:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2686.149369] Lustre: server umount lustre-MDT0001 complete [ 2689.910435] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2691.032270] Key type lgssc unregistered [ 2691.158415] LNet: 161666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.161455] LNetError: 161666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.171500] LNet: Removed LNI 192.168.204.121@tcp [ 2691.472729] Key type .llcrypt unregistered [ 2691.473785] Key type ._llcrypt unregistered [ 2703.406792] Key type ._llcrypt registered [ 2703.408612] Key type .llcrypt registered [ 2703.453202] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2703.855985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2703.915070] alg: No test for adler32 (adler32-zlib) [ 2704.777975] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2704.867406] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2706.463229] Key type lgssc registered [ 2706.877987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2709.660917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2709.670038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.844535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.311647] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2715.299138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2715.434120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.865508] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2718.649091] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.954094] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.334224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2722.467405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.402770] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2726.319239] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2735.782896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2741.216617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.218634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.220118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.222267] Lustre: Skipped 1 previous similar message [ 2744.366202] LustreError: 165590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2744.429599] Lustre: server umount lustre-OST0000 complete [ 2746.336868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.336975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.337824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.337829] Lustre: Skipped 1 previous similar message [ 2746.339756] Lustre: Skipped 1 previous similar message [ 2746.349128] LustreError: Skipped 1 previous similar message [ 2751.456733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.458284] Lustre: Skipped 2 previous similar messages [ 2751.808548] LustreError: 165791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2751.811533] LustreError: 165791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2751.837360] Lustre: server umount lustre-MDT0000 complete [ 2753.186283] LustreError: 163305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826369 with bad export cookie 9330145475773335960 [ 2753.189652] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.190115] LustreError: 163305:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2753.256382] LustreError: 165993:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2753.258148] LustreError: 165993:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2753.348286] Lustre: server umount lustre-MDT0001 complete [ 2756.930184] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2758.152319] Key type lgssc unregistered [ 2758.286377] LNet: 166524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2758.288373] LNetError: 166524:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2758.295462] LNet: Removed LNI 192.168.204.121@tcp [ 2758.618262] Key type .llcrypt unregistered [ 2758.619265] Key type ._llcrypt unregistered [ 2767.085796] Key type ._llcrypt registered [ 2767.086824] Key type .llcrypt registered [ 2767.121466] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2767.431448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2767.442269] alg: No test for adler32 (adler32-zlib) [ 2768.287102] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2768.365266] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2769.943143] Key type lgssc registered [ 2770.343967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2774.060440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2774.067609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2775.243440] Lustre: *** cfs_fail_loc=135, val=0*** [ 2775.288784] LustreError: 168005:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2775.292478] LustreError: 168005:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.121@tcp: cfg command failed: rc = -2 [ 2775.297139] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2775.297139] [ 2775.302095] LustreError: MGC192.168.204.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2775.306605] LustreError: 167964:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2775.310060] LustreError: 167964:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2775.312244] LustreError: 167964:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2775.315085] LustreError: 167964:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2775.385244] Lustre: server umount lustre-MDT0000 complete [ 2775.386732] LustreError: 167964:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2777.532077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.707575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.056585] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2781.840818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2781.961487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.244713] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2784.857157] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2786.016487] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2786.657821] LustreError: 168408:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2786.660444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.660870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.661094] LustreError: 168408:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2786.666582] Lustre: Skipped 1 previous similar message [ 2786.668223] LustreError: 168408:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2791.776568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.778297] Lustre: Skipped 2 previous similar messages [ 2792.762795] LustreError: 169719:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2792.764854] LustreError: 169719:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2792.791300] Lustre: server umount lustre-MDT0000 complete [ 2794.100423] LustreError: 168358:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826410 with bad export cookie 17944634754825448445 [ 2794.104642] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.106168] LustreError: 168358:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2794.168304] LustreError: 169922:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2794.171279] LustreError: 169922:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2794.260991] Lustre: server umount lustre-MDT0001 complete [ 2797.902971] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2799.031500] Key type lgssc unregistered [ 2799.171376] LNet: 170455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2799.175173] LNetError: 170455:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2799.182331] LNet: Removed LNI 192.168.204.121@tcp [ 2799.490364] Key type .llcrypt unregistered [ 2799.492253] Key type ._llcrypt unregistered [ 2806.381712] Key type ._llcrypt registered [ 2806.382970] Key type .llcrypt registered [ 2806.421585] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2806.527239] Key type .llcrypt unregistered [ 2806.528226] Key type ._llcrypt unregistered [ 2813.339351] Key type ._llcrypt registered [ 2813.340925] Key type .llcrypt registered [ 2813.377490] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2813.716379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.765205] alg: No test for adler32 (adler32-zlib) [ 2814.613872] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2814.699994] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2816.279138] Key type lgssc registered [ 2816.656838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2819.280485] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2819.287053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2834.719309] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2839.839169] LustreError: 172523:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.121@tcp: failed processing log, type 4: rc = -110 [ 2866.463186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.243152] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2869.997890] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2872.293821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2872.473294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.498729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2873.770951] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2876.597986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2876.756316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.085911] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2879.827904] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2880.990565] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2881.965972] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2891.235955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.238150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.239481] Lustre: Skipped 1 previous similar message [ 2895.533382] LustreError: 175247:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2895.593557] Lustre: server umount lustre-OST0000 complete [ 2897.375921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2897.378829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2897.383154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2897.384974] Lustre: Skipped 1 previous similar message [ 2901.472220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.472732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.592187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.594099] Lustre: Skipped 2 previous similar messages [ 2911.199102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2911.232138] LustreError: 175449:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2911.235388] LustreError: 175449:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2911.272269] Lustre: server umount lustre-MDT0000 complete [ 2911.712342] LustreError: 173450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.720067] LustreError: 173450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2912.528807] LustreError: 172530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826529 with bad export cookie 3232610993117234063 [ 2912.533491] LustreError: 172530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2912.534226] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.597827] LustreError: 175653:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2912.600652] LustreError: 175653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2912.701710] Lustre: server umount lustre-MDT0001 complete [ 2916.331921] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 2917.395365] Key type lgssc unregistered [ 2917.538460] LNet: 176183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2917.540871] LNetError: 176183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2917.548302] LNet: Removed LNI 192.168.204.121@tcp [ 2917.890710] Key type .llcrypt unregistered [ 2917.891844] Key type ._llcrypt unregistered [ 2926.530245] Key type ._llcrypt registered [ 2926.531827] Key type .llcrypt registered [ 2926.575228] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 2927.082605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2927.093256] alg: No test for adler32 (adler32-zlib) [ 2927.967198] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 2928.065455] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 2929.655198] Key type lgssc registered [ 2930.014429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2932.665184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2932.671781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.829225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.065738] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2937.791443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2937.912193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2939.172552] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2940.908827] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2942.127986] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2944.447470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2944.575978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.483558] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2948.299625] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2958.947545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2964.980334] Lustre: Failing over lustre-MDT0000 [ 2965.053122] LustreError: 179872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2965.089731] Lustre: server umount lustre-MDT0000 complete [ 2968.544422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.547981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.552139] LustreError: 177553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.556566] LustreError: 177553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2969.568322] LustreError: 177554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.569775] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.572500] LustreError: 177554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2969.578051] Lustre: Skipped 1 previous similar message [ 2974.021828] LustreError: 179715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2974.029392] LustreError: 179715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2978.004481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.045535] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.137341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.177354] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2979.142058] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2979.427229] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 2980.963944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2983.401583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2983.406676] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2983.426907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2988.025300] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2998.755281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.759544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.759587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3003.438094] LustreError: 181252:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3003.440011] LustreError: 181252:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3003.503885] Lustre: server umount lustre-OST0000 complete [ 3008.994172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.994553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3008.995693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.995699] Lustre: Skipped 1 previous similar message [ 3008.997456] Lustre: Skipped 2 previous similar messages [ 3014.112212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.114131] Lustre: Skipped 2 previous similar messages [ 3019.231152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.232722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.235737] Lustre: Skipped 1 previous similar message [ 3019.266286] LustreError: 181453:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3019.269326] LustreError: 181453:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3019.294937] Lustre: server umount lustre-MDT0000 complete [ 3020.549279] LustreError: 179174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826637 with bad export cookie 14721708569773758046 [ 3020.552492] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.552741] LustreError: 179174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3020.689069] Lustre: server umount lustre-MDT0001 complete [ 3024.150809] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3025.170889] Key type lgssc unregistered [ 3025.288310] LNet: 182186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3025.290545] LNetError: 182186:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.297319] LNet: Removed LNI 192.168.204.121@tcp [ 3025.595085] Key type .llcrypt unregistered [ 3025.596861] Key type ._llcrypt unregistered [ 3034.475473] Key type ._llcrypt registered [ 3034.477164] Key type .llcrypt registered [ 3034.526860] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3035.042139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3035.052285] alg: No test for adler32 (adler32-zlib) [ 3035.922323] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3036.014125] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3037.599126] Key type lgssc registered [ 3037.965477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3040.708813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3040.716965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.880500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.237622] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3046.176814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3046.301913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.657390] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3049.483720] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3050.757764] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3053.046236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3053.173815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.024741] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3056.872563] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3066.471275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3067.780345] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3071.199558] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3079.182752] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3092.453188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.454454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.457302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.460217] Lustre: Skipped 1 previous similar message [ 3094.445099] LustreError: 185994:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3094.520448] Lustre: server umount lustre-OST0000 complete [ 3097.569697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3097.570772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.571541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.571550] Lustre: Skipped 1 previous similar message [ 3097.571593] LustreError: Skipped 1 previous similar message [ 3097.577013] Lustre: Skipped 1 previous similar message [ 3102.687955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.690081] Lustre: Skipped 2 previous similar messages [ 3107.808683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.810166] Lustre: Skipped 1 previous similar message [ 3110.367108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3110.409724] LustreError: 186195:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3110.412217] LustreError: 186195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3110.443497] Lustre: server umount lustre-MDT0000 complete [ 3111.690141] LustreError: 183539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826728 with bad export cookie 2799013332123846664 [ 3111.692822] LustreError: 183539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3111.693502] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.758818] LustreError: 186398:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3111.762132] LustreError: 186398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3111.856928] Lustre: server umount lustre-MDT0001 complete [ 3115.398765] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3116.523079] Key type lgssc unregistered [ 3116.659482] LNet: 186928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3116.661491] LNetError: 186928:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3116.670541] LNet: Removed LNI 192.168.204.121@tcp [ 3116.987828] Key type .llcrypt unregistered [ 3116.988883] Key type ._llcrypt unregistered [ 3125.399625] Key type ._llcrypt registered [ 3125.400713] Key type .llcrypt registered [ 3125.443994] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3125.846533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3125.854386] alg: No test for adler32 (adler32-zlib) [ 3126.704788] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3126.784482] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3128.359128] Key type lgssc registered [ 3128.745567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3131.231102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3131.238159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.384924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.614797] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3136.377850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3136.497663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3137.830300] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3139.595738] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3140.753287] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3142.887031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3143.004318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.842855] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3146.669893] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3157.350881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3163.860953] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3175.693809] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3181.304725] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3191.291570] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3198.435149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3198.435530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.436719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3198.437324] LustreError: Skipped 1 previous similar message [ 3198.441644] Lustre: Skipped 1 previous similar message [ 3203.552471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.555158] Lustre: Skipped 2 previous similar messages [ 3204.467136] LustreError: 192491:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3204.562138] Lustre: server umount lustre-OST0000 complete [ 3208.160700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3208.166485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.175768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.240537] LustreError: 192692:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3212.245468] LustreError: 192692:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3212.288017] Lustre: server umount lustre-MDT0000 complete [ 3213.792246] LustreError: 188300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.799304] LustreError: 188300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3214.003206] LustreError: 188281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826830 with bad export cookie 2390509704109060807 [ 3214.006331] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3214.006673] LustreError: 188281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3214.069171] LustreError: 192893:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3214.070960] LustreError: 192893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3214.149066] Lustre: server umount lustre-MDT0001 complete [ 3218.269413] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3219.575712] Key type lgssc unregistered [ 3219.726386] LNet: 193424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3219.731243] LNetError: 193424:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3219.740475] LNet: Removed LNI 192.168.204.121@tcp [ 3220.106918] Key type .llcrypt unregistered [ 3220.108169] Key type ._llcrypt unregistered [ 3230.061602] Key type ._llcrypt registered [ 3230.062563] Key type .llcrypt registered [ 3230.113701] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3230.558403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3230.589100] alg: No test for adler32 (adler32-zlib) [ 3231.463956] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3231.563168] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3233.151281] Key type lgssc registered [ 3233.794884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3237.457366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3237.473714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.684380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.242146] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3243.661669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3243.867894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3245.352198] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3247.550252] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3249.058616] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3251.511055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3251.657469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3254.184829] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3256.408096] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3263.981209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3274.214009] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.214588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3274.222916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3274.225653] LustreError: Skipped 1 previous similar message [ 3274.231319] Lustre: Skipped 1 previous similar message [ 3276.531151] LustreError: 197241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3276.615825] Lustre: server umount lustre-OST0000 complete [ 3279.329946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.337293] Lustre: Skipped 1 previous similar message [ 3279.341122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.344877] Lustre: Skipped 1 previous similar message [ 3284.046758] LustreError: 197448:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3284.051422] LustreError: 197448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3284.084938] Lustre: server umount lustre-MDT0000 complete [ 3284.448616] LustreError: 196985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.458385] LustreError: 196985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3285.934525] LustreError: 194837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826902 with bad export cookie 6528236015740569044 [ 3285.940334] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.942330] LustreError: 194837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3286.030829] LustreError: 197652:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3286.033180] LustreError: 197652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3286.124694] Lustre: server umount lustre-MDT0001 complete [ 3290.644170] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3291.828581] Key type lgssc unregistered [ 3291.984619] LNet: 198184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3291.987546] LNetError: 198184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3291.995566] LNet: Removed LNI 192.168.204.121@tcp [ 3292.444747] Key type .llcrypt unregistered [ 3292.447114] Key type ._llcrypt unregistered [ 3300.049272] Key type ._llcrypt registered [ 3300.050423] Key type .llcrypt registered [ 3300.087748] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3300.387510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3300.397759] alg: No test for adler32 (adler32-zlib) [ 3301.250889] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3301.332317] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3302.911211] Key type lgssc registered [ 3303.347523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.003146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.008922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3307.163937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3308.520100] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3311.640917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3311.775866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3313.070503] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3315.009875] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3316.238225] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3318.438726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3318.568003] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.335704] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3322.097447] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3331.878623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3337.186187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3337.186708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3337.192482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3337.194533] Lustre: Skipped 1 previous similar message [ 3337.836131] LustreError: 201826:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3337.904916] Lustre: server umount lustre-OST0000 complete [ 3341.179880] LustreError: 202027:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3341.181946] LustreError: 202027:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3341.207238] Lustre: server umount lustre-MDT0000 complete [ 3342.305285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.306373] LustreError: 200725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.309433] Lustre: Skipped 2 previous similar messages [ 3342.316505] LustreError: 200725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3342.423470] LustreError: 201740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758826959 with bad export cookie 6262077061992534380 [ 3342.429624] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3342.495790] LustreError: 202228:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3342.497872] LustreError: 202228:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3342.576357] Lustre: server umount lustre-MDT0001 complete [ 3346.374062] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3347.417282] Key type lgssc unregistered [ 3347.554337] LNet: 202758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3347.556762] LNetError: 202758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3347.568287] LNet: Removed LNI 192.168.204.121@tcp [ 3347.857592] Key type .llcrypt unregistered [ 3347.858877] Key type ._llcrypt unregistered [ 3354.972804] Key type ._llcrypt registered [ 3354.973769] Key type .llcrypt registered [ 3355.005015] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3355.417388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3355.426056] alg: No test for adler32 (adler32-zlib) [ 3356.273924] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3356.350566] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3357.935163] Key type lgssc registered [ 3358.279851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.803813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.810023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.960398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.335975] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3366.147161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3366.283961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.514942] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3369.314610] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3370.466468] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3372.715212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3372.840160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3374.853789] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3376.745978] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3387.173972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3389.897638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3389.926441] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3390.063277] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3392.057879] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3394.204228] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3399.844340] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 3410.403022] Lustre: Permanently deactivating lustre-OST0001 [ 3410.410471] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3418.105107] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3422.317384] Lustre: Permanently reactivating lustre-OST0001 [ 3422.322320] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3422.324180] Lustre: Skipped 2 previous similar messages [ 3424.246343] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.250083] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3424.252269] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3424.257703] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3424.268569] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3431.396181] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.397312] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3431.402198] Lustre: Skipped 2 previous similar messages [ 3431.407467] Lustre: Skipped 1 previous similar message [ 3436.517401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3436.529119] LustreError: 207472:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3436.547875] Lustre: server umount lustre-OST0001 complete [ 3438.560443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.562679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.566306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3438.568660] Lustre: Skipped 1 previous similar message [ 3441.635102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3441.637092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.637944] LustreError: 207474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.649194] LustreError: 207474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3446.755803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.756497] LustreError: 205579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.757363] Lustre: Skipped 2 previous similar messages [ 3446.761063] LustreError: 205579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3451.877489] LustreError: 205581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.882204] LustreError: 205581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3452.383091] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.404455] LustreError: 207674:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3452.407442] LustreError: 207674:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3452.483386] Lustre: server umount lustre-OST0000 complete [ 3456.993712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.994763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3456.996792] Lustre: Skipped 1 previous similar message [ 3456.999120] Lustre: Skipped 3 previous similar messages [ 3468.255085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3468.287542] LustreError: 207876:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3468.288956] LustreError: 207876:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3468.317500] Lustre: server umount lustre-MDT0000 complete [ 3469.659401] LustreError: 203963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827086 with bad export cookie 9031623006032680887 [ 3469.662724] LustreError: 203963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3469.664038] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.839383] Lustre: server umount lustre-MDT0001 complete [ 3473.681738] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3474.945807] Key type lgssc unregistered [ 3475.080427] LNet: 208609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.083349] LNetError: 208609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3475.092520] LNet: Removed LNI 192.168.204.121@tcp [ 3475.511263] Key type .llcrypt unregistered [ 3475.512324] Key type ._llcrypt unregistered [ 3487.110572] Key type ._llcrypt registered [ 3487.111728] Key type .llcrypt registered [ 3487.158452] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3493.122507] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3493.530353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3493.543191] alg: No test for adler32 (adler32-zlib) [ 3494.385940] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3494.460164] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3496.039136] Key type lgssc registered [ 3496.390878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3499.454364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.683718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3503.742094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3505.967361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3508.358769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3508.381927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3508.387523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.470893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3509.481966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3509.523253] Lustre: lustre-MDT0000: new disk, initializing [ 3509.547079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.553896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3510.801896] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3515.515254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3515.545981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3515.587399] Lustre: 212526:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3515.611343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3515.613547] Lustre: Skipped 1 previous similar message [ 3515.652521] Lustre: lustre-MDT0001: new disk, initializing [ 3515.690709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.704555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3515.707626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3517.116437] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3520.440801] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3521.648259] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3523.945389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3523.975905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3524.091263] Lustre: lustre-OST0000: new disk, initializing [ 3524.093153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3524.127094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.027121] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3529.162252] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3530.601510] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3533.303899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3533.307649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3533.336517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3533.732097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3534.960661] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3535.023750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3538.401065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3538.401552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.403126] LustreError: Skipped 1 previous similar message [ 3538.403762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3538.406479] Lustre: Skipped 1 previous similar message [ 3541.677114] LustreError: 215489:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3541.738277] Lustre: server umount lustre-OST0000 complete [ 3543.522868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3543.524053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3543.526143] Lustre: Skipped 1 previous similar message [ 3543.532052] Lustre: Skipped 2 previous similar messages [ 3548.640517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.642952] Lustre: Skipped 1 previous similar message [ 3553.760777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3553.762566] Lustre: Skipped 1 previous similar message [ 3557.343156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3557.386197] LustreError: 215690:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3557.389590] LustreError: 215690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3557.426437] Lustre: server umount lustre-MDT0000 complete [ 3558.785140] LustreError: 212518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827175 with bad export cookie 18249135729568218538 [ 3558.790264] LustreError: 212518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3558.790826] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.879435] LustreError: 215894:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3558.882475] LustreError: 215894:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3558.999315] Lustre: server umount lustre-MDT0001 complete [ 3561.953320] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 15:06:18 (1758827178) [ 3564.158488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.367446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.679819] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3568.389132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3568.500640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.916426] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3571.708917] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3572.916683] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3575.246080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3577.213027] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3578.983136] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3595.216913] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3597.648222] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3605.226521] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3615.009888] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3623.625057] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3642.122474] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3642.125889] Lustre: Skipped 2 previous similar messages [ 3675.694448] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3675.696651] Lustre: Skipped 3 previous similar messages [ 3741.111226] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3741.116423] Lustre: Skipped 8 previous similar messages [ 3752.929771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.933960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.941401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.955211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3755.629581] LustreError: 220066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3755.632064] LustreError: 220066:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3755.688762] Lustre: server umount lustre-OST0000 complete [ 3758.047828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3758.049920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.053078] Lustre: Skipped 1 previous similar message [ 3764.193641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3764.197716] Lustre: Skipped 6 previous similar messages [ 3771.359089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3771.390506] LustreError: 220271:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3771.392609] LustreError: 220271:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3771.423126] Lustre: server umount lustre-MDT0000 complete [ 3772.704502] LustreError: 217282:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827389 with bad export cookie 18249135729568219406 [ 3772.707738] LustreError: 217282:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3772.708096] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3772.854039] Lustre: server umount lustre-MDT0001 complete [ 3776.362268] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3777.332930] Key type lgssc unregistered [ 3777.473421] LNet: 221004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3777.477207] LNetError: 221004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3777.484486] LNet: Removed LNI 192.168.204.121@tcp [ 3777.801930] Key type .llcrypt unregistered [ 3777.803317] Key type ._llcrypt unregistered [ 3786.898526] Key type ._llcrypt registered [ 3786.899672] Key type .llcrypt registered [ 3786.940141] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3787.300983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3787.342087] alg: No test for adler32 (adler32-zlib) [ 3788.205951] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3788.288237] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3789.863137] Key type lgssc registered [ 3790.228459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3792.763906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3792.770192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.929568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.240576] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3798.260783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3798.387409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3799.719671] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3801.532100] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3802.788247] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3805.543344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3805.712662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.710580] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_default_debug -1 all [ 3809.517686] Lustre: DEBUG MARKER: oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3829.221489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.221885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.230621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.232867] LustreError: Skipped 1 previous similar message [ 3829.238506] Lustre: Skipped 1 previous similar message [ 3832.177272] LustreError: 224795:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3832.259382] Lustre: server umount lustre-OST0000 complete [ 3834.338281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3834.338624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.339824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.345709] Lustre: Skipped 2 previous similar messages [ 3839.456725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.458350] Lustre: Skipped 2 previous similar messages [ 3839.613504] LustreError: 225002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3839.615315] LustreError: 225002:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3839.641217] Lustre: server umount lustre-MDT0000 complete [ 3841.019979] LustreError: 223092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1758827457 with bad export cookie 17997731702410795186 [ 3841.025676] LustreError: MGC192.168.204.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3841.028056] LustreError: 223092:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3841.110075] LustreError: 225204:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3841.113103] LustreError: 225204:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3841.203983] Lustre: server umount lustre-MDT0001 complete [ 3844.734221] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3845.986457] Key type lgssc unregistered [ 3846.130504] LNet: 225745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3846.133273] LNetError: 225745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3846.142381] LNet: Removed LNI 192.168.204.121@tcp [ 3846.484795] Key type .llcrypt unregistered [ 3846.485861] Key type ._llcrypt unregistered [ 3853.520517] Key type ._llcrypt registered [ 3853.521533] Key type .llcrypt registered [ 3853.556416] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3853.648450] Key type .llcrypt unregistered [ 3853.649571] Key type ._llcrypt unregistered [ 3861.770120] Key type ._llcrypt registered [ 3861.771095] Key type .llcrypt registered [ 3861.810172] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing set_hostid [ 3868.000153] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing load_modules_local [ 3868.318852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.371027] alg: No test for adler32 (adler32-zlib) [ 3869.235412] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 3869.311129] LNet: Added LNI 192.168.204.121@tcp [8/256/0/180] [ 3870.887211] Key type lgssc registered [ 3871.232284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3874.910573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3877.731021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3879.836283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3882.194762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3882.789023] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3751 sec ========== 15:11:38 (1758827498) [ 3883.367275] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:11:39 (1758827499) === [ 3884.504118] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:11:40 (1758827500) === [ 3899.691477] Lustre: DEBUG MARKER: oleg421-server.virtnet: executing unload_modules_local [ 3900.917958] Key type lgssc unregistered [ 3901.056295] LNet: 231821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3901.058486] LNetError: 231821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3901.067451] LNet: Removed LNI 192.168.204.121@tcp [ 3901.388396] Key type .llcrypt unregistered [ 3901.390202] Key type ._llcrypt unregistered