[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 632005821 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003244] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.006010] kvm-guest: setup PV IPIs [ 0.007895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009000] pid_max: default: 32768 minimum: 301 [ 0.009125] LSM: Security Framework initializing [ 0.010067] Yama: becoming mindful. [ 0.011054] SELinux: Initializing. [ 0.013188] *** VALIDATE selinux *** [ 0.024523] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029539] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030195] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032191] *** VALIDATE tmpfs *** [ 0.033196] *** VALIDATE proc *** [ 0.034303] *** VALIDATE cgroup *** [ 0.035013] *** VALIDATE cgroup2 *** [ 0.037441] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038249] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041039] Spectre V2 : User space: Vulnerable [ 0.042031] Speculative Store Bypass: Vulnerable [ 0.045640] debug: unmapping init [mem 0xffffffff9ec59000-0xffffffff9ec60fff] [ 0.047312] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048764] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050018] ... bit width: 48 [ 0.051025] ... generic registers: 4 [ 0.052030] ... value mask: 0000ffffffffffff [ 0.053020] ... max period: 00007fffffffffff [ 0.054026] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056489] rcu: Hierarchical SRCU implementation. [ 0.058636] smp: Bringing up secondary CPUs ... [ 0.060046] x86: Booting SMP configuration: [ 0.061012] .... node #0, CPUs: #1 #2 #3 [ 0.083141] smp: Brought up 1 node, 4 CPUs [ 0.085027] smpboot: Max logical packages: 1 [ 0.086157] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.164000] node 0 deferred pages initialised in 65ms [ 0.171268] devtmpfs: initialized [ 0.174199] x86/mm: Memory block size: 128MB [ 0.178686] gcov: version magic: 0x41383552 [ 0.183086] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.187108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.190391] pinctrl core: initialized pinctrl subsystem [ 0.194360] [ 0.195011] ************************************************************* [ 0.198015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199014] ** ** [ 0.202022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.205022] ** ** [ 0.208017] ** This means that this kernel is built to expose internal ** [ 0.211017] ** IOMMU data structures, which may compromise security on ** [ 0.214069] ** your system. ** [ 0.217021] ** ** [ 0.220024] ** If you see this message and you are not debugging the ** [ 0.225018] ** kernel, report this immediately to your vendor! ** [ 0.229022] ** ** [ 0.237018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.238068] ************************************************************* [ 0.243093] NET: Registered protocol family 16 [ 0.244935] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.248086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.251097] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.255563] cpuidle: using governor menu [ 0.257000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.260023] PCI: Using configuration type 1 for base access [ 0.261143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.272157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.273026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.275245] cryptd: max_cpu_qlen set to 1000 [ 0.279384] ACPI: Added _OSI(Module Device) [ 0.281017] ACPI: Added _OSI(Processor Device) [ 0.283017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.285017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289587] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297430] ACPI: Interpreter enabled [ 0.298000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.298000] ACPI: Using IOAPIC for interrupt routing [ 0.298000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.302757] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.314013] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.316057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.318023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.321200] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.326997] acpiphp: Slot [2] registered [ 0.328564] acpiphp: Slot [3] registered [ 0.331153] acpiphp: Slot [4] registered [ 0.332122] acpiphp: Slot [5] registered [ 0.334221] acpiphp: Slot [6] registered [ 0.335266] acpiphp: Slot [7] registered [ 0.337270] acpiphp: Slot [8] registered [ 0.339170] acpiphp: Slot [9] registered [ 0.340156] acpiphp: Slot [10] registered [ 0.342247] acpiphp: Slot [11] registered [ 0.344324] acpiphp: Slot [12] registered [ 0.346345] acpiphp: Slot [13] registered [ 0.348153] acpiphp: Slot [14] registered [ 0.350138] acpiphp: Slot [15] registered [ 0.352143] acpiphp: Slot [16] registered [ 0.353133] acpiphp: Slot [17] registered [ 0.355153] acpiphp: Slot [18] registered [ 0.357135] acpiphp: Slot [19] registered [ 0.358366] acpiphp: Slot [20] registered [ 0.360150] acpiphp: Slot [21] registered [ 0.362140] acpiphp: Slot [22] registered [ 0.364120] acpiphp: Slot [23] registered [ 0.365157] acpiphp: Slot [24] registered [ 0.367157] acpiphp: Slot [25] registered [ 0.368103] acpiphp: Slot [26] registered [ 0.369103] acpiphp: Slot [27] registered [ 0.371132] acpiphp: Slot [28] registered [ 0.373188] acpiphp: Slot [29] registered [ 0.374164] acpiphp: Slot [30] registered [ 0.376123] acpiphp: Slot [31] registered [ 0.377082] PCI host bridge to bus 0000:00 [ 0.380031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.382028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.386045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389044] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393035] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.394045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.398719] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.406339] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.412211] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.424021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.429065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.434026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.435000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.437031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.440927] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.444075] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.447051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.451078] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.457021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.469970] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.477029] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485305] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.495033] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505044] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.528025] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.547701] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.579028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.600028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.621031] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.632075] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.639027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.647023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.667021] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.682179] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.688023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.696026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.713021] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.726565] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.734024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.740109] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.756000] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.769148] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.778026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.781029] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.794029] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.806000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.809209] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.813064] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.815457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.823300] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.844014] iommu: Default domain type: Passthrough [ 0.849150] SCSI subsystem initialized [ 0.851228] ACPI: bus type USB registered [ 0.854152] usbcore: registered new interface driver usbfs [ 0.858184] usbcore: registered new interface driver hub [ 0.860131] usbcore: registered new device driver usb [ 0.866142] pps_core: LinuxPPS API ver. 1 registered [ 0.869017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.876097] PTP clock support registered [ 0.879083] EDAC MC: Ver: 3.0.0 [ 0.880403] PCI: Using ACPI for IRQ routing [ 0.883783] NetLabel: Initializing [ 0.884011] NetLabel: domain hash size = 128 [ 0.887016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.889119] NetLabel: unlabeled traffic allowed by default [ 0.891381] vgaarb: loaded [ 0.892279] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.894014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.904241] clocksource: Switched to clocksource kvm-clock [ 1.091870] VFS: Disk quotas dquot_6.6.0 [ 1.095753] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.098568] *** VALIDATE ramfs *** [ 1.100121] *** VALIDATE hugetlbfs *** [ 1.101896] pnp: PnP ACPI init [ 1.105721] pnp: PnP ACPI: found 6 devices [ 1.128906] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.134319] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.136612] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.138576] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.140384] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.142856] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.146679] NET: Registered protocol family 2 [ 1.151207] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.159713] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.167108] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.172576] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.175900] TCP: Hash tables configured (established 65536 bind 65536) [ 1.183578] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.186608] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.189477] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.192822] NET: Registered protocol family 1 [ 1.196860] RPC: Registered named UNIX socket transport module. [ 1.203551] RPC: Registered udp transport module. [ 1.205522] RPC: Registered tcp transport module. [ 1.207422] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.210257] NET: Registered protocol family 44 [ 1.211828] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.214332] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.216847] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.220539] PCI: CLS 0 bytes, default 64 [ 1.224987] Unpacking initramfs... [ 3.425958] debug: unmapping init [mem 0xffff9a6d7cc54000-0xffff9a6d7ffbffff] [ 3.433031] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.435753] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.439870] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.104657] Initialise system trusted keyrings [ 4.106572] Key type blacklist registered [ 4.113051] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.121930] zbud: loaded [ 4.125436] *** VALIDATE nfs *** [ 4.126832] *** VALIDATE nfs4 *** [ 4.132829] pstore: using deflate compression [ 4.136642] Platform Keyring initialized [ 4.337398] NET: Registered protocol family 38 [ 4.342514] Key type asymmetric registered [ 4.343896] Asymmetric key parser 'x509' registered [ 4.352328] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.360453] io scheduler mq-deadline registered [ 4.362577] io scheduler kyber registered [ 4.375670] io scheduler bfq registered [ 4.378200] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.381315] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.384079] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.388129] ACPI: Power Button [PWRF] [ 4.494726] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.660172] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.209821] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.430493] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.916378] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.997185] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.094679] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.105144] Non-volatile memory driver v1.3 [ 6.106780] Linux agpgart interface v0.103 [ 6.243944] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 6.247481] vda: detected capacity change from 0 to 68526080 [ 6.294095] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.305341] vdb: detected capacity change from 0 to 1073741824 [ 6.336819] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.342989] vdc: detected capacity change from 0 to 2621440000 [ 6.373320] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.382626] vdd: detected capacity change from 0 to 2621440000 [ 6.410908] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.417676] vde: detected capacity change from 0 to 4294967296 [ 6.438630] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.441347] vdf: detected capacity change from 0 to 4294967296 [ 6.472228] libphy: Fixed MDIO Bus: probed [ 6.490842] usbcore: registered new interface driver usbserial_generic [ 6.497656] usbserial: USB Serial support registered for generic [ 6.500173] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.524986] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.531120] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.537108] mousedev: PS/2 mouse device common for all mice [ 6.544401] rtc_cmos 00:05: RTC can wake from S4 [ 6.551176] rtc_cmos 00:05: registered as rtc0 [ 6.557844] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.561271] intel_pstate: CPU model not supported [ 6.569975] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.582196] hid: raw HID events driver (C) Jiri Kosina [ 6.584063] usbcore: registered new interface driver usbhid [ 6.589179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.600800] usbhid: USB HID core driver [ 6.601791] drop_monitor: Initializing network drop monitor service [ 6.601929] Initializing XFRM netlink socket [ 6.602292] NET: Registered protocol family 10 [ 6.624537] Segment Routing with IPv6 [ 6.626107] NET: Registered protocol family 17 [ 6.628744] mpls_gso: MPLS GSO support [ 6.652207] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.663237] RAS: Correctable Errors collector initialized. [ 6.665434] AVX version of gcm_enc/dec engaged. [ 6.667542] AES CTR mode by8 optimization enabled [ 6.892865] sched_clock: Marking stable (6892844484, 0)->(8333389125, -1440544641) [ 6.924795] registered taskstats version 1 [ 6.934709] Loading compiled-in X.509 certificates [ 6.936682] zswap: loaded using pool lzo/zbud [ 7.003807] Key type big_key registered [ 7.028158] Key type encrypted registered [ 7.030706] ima: No TPM chip found, activating TPM-bypass! [ 7.034825] ima: Allocated hash algorithm: sha1 [ 7.038683] ima: No architecture policies found [ 7.042591] evm: Initialising EVM extended attributes: [ 7.045971] evm: security.selinux [ 7.048083] evm: security.ima [ 7.048942] evm: security.capability [ 7.053231] evm: HMAC attrs: 0x1 [ 7.057426] rtc_cmos 00:05: setting system clock to 2025-11-16 22:25:02 UTC (1763331902) [ 7.072334] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 7.080339] debug: unmapping init [mem 0xffffffff9e982000-0xffffffff9ec58fff] [ 7.091185] Write protecting the kernel read-only data: 28672k [ 7.097525] debug: unmapping init [mem 0xffffffff9d003000-0xffffffff9d1fffff] [ 7.106694] debug: unmapping init [mem 0xffffffff9d914000-0xffffffff9d9fffff] [ 7.200601] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.230491] systemd[1]: Detected virtualization kvm. [ 7.232695] systemd[1]: Detected architecture x86-64. [ 7.243568] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.275854] systemd[1]: No hostname configured. [ 7.278635] systemd[1]: Set hostname to . [ 7.284435] random: systemd: uninitialized urandom read (16 bytes read) [ 7.286790] systemd[1]: Initializing machine ID from random generator. [ 7.619590] random: systemd: uninitialized urandom read (16 bytes read) [ 7.622052] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.634970] random: systemd: uninitialized urandom read (16 bytes read) [ 7.638535] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.651731] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.125097] device-mapper: uevent: version 1.0.3 [ 9.134831] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.533506] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 11.660269] random: fast init done [ 12.016765] scsi host0: ata_piix [ 12.183749] scsi host1: ata_piix [ 12.185041] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.187210] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.577981] random: crng init done [ 16.580909] random: 7 urandom warning(s) missed due to ratelimiting [ 19.832550] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.024301] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.763334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.164690] SELinux: Disabled at runtime. [ 28.360098] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.428504] systemd[1]: Detected virtualization kvm. [ 28.430597] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.945634] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.948896] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.977832] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.988868] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.014295] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.047807] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.160729] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ 31.870622] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 33.697466] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.220246] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.452564] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.309773] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.794993] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 43.197749] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 43.760800] NFS: Registering the id_resolver key type [ 43.762759] Key type id_resolver registered [ 43.764908] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ 57.467982] hrtimer: interrupt took 7971914 ns [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg219-server login: [ 95.476151] libcfs: loading out-of-tree module taints kernel. [ 95.493144] Key type ._llcrypt registered [ 95.494378] Key type .llcrypt registered [ 95.575709] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 108.419206] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 109.360887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.375030] alg: No test for adler32 (adler32-zlib) [ 110.928723] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 111.841356] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 113.559134] Key type lgssc registered [ 114.993479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.917558] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.823080] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.187829] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.752088] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 158.945573] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.686922] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 194.827757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.872800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.884130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.211956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.275425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.408955] Lustre: lustre-MDT0000: new disk, initializing [ 196.532456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.599295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.309657] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.216176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.298826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 212.338485] Lustre: 6522:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.366090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.368035] Lustre: Skipped 1 previous similar message [ 212.422183] Lustre: lustre-MDT0001: new disk, initializing [ 212.484776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.514118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 212.519703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.437951] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.506506] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.704991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.759295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 227.960419] Lustre: lustre-OST0000: new disk, initializing [ 227.963223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.997568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.962627] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.495332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 233.504058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 233.544189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.705603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 244.752138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 244.838376] Lustre: lustre-OST0001: new disk, initializing [ 244.840780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.916611] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.223339] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.467290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.472074] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.517571] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.334586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.607700] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 277.381132] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.507923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing yml_node [ 285.749244] Lustre: DEBUG MARKER: Client: 2.16.61.53 [ 287.825807] Lustre: DEBUG MARKER: MDS: 2.16.61.53 [ 290.333539] Lustre: DEBUG MARKER: OSS: 2.16.61.53 [ 291.748306] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 17:29:45 EST 2025 [ 306.665134] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 307.879087] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 325.087878] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.089246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.103776] Lustre: Skipped 1 previous similar message [ 325.110986] Lustre: Skipped 2 previous similar messages [ 330.207763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.212212] Lustre: Skipped 5 previous similar messages [ 335.333815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.336720] Lustre: Skipped 3 previous similar messages [ 336.351156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.445703] LustreError: 12225:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 336.533603] Lustre: server umount lustre-MDT0000 complete [ 339.768721] LustreError: 6513:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332235 with bad export cookie 8805660048754981959 [ 339.782236] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.449902] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.459602] Lustre: Skipped 1 previous similar message [ 340.473186] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.475895] Lustre: Skipped 1 previous similar message [ 344.480221] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.484744] Lustre: Skipped 1 previous similar message [ 349.602857] LustreError: 6528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.614829] LustreError: 6528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 354.271901] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.359442] LustreError: 12428:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 354.364812] LustreError: 12428:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 354.527279] Lustre: server umount lustre-MDT0001 complete [ 358.144771] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 358.147755] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 358.192198] Lustre: server umount lustre-OST0000 complete [ 361.751334] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 361.759619] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 361.863447] Lustre: server umount lustre-OST0001 complete [ 366.838577] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 373.317756] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 381.820238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.205594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.923118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 401.313110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 407.756990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 407.816157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.055299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 408.096547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 408.185695] Lustre: lustre-MDT0000: new disk, initializing [ 408.307201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.334054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.018602] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 422.652266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 422.747613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 422.846853] Lustre: 15016:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 422.989418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.013925] Lustre: Skipped 1 previous similar message [ 423.156507] Lustre: lustre-MDT0001: new disk, initializing [ 423.266995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.309299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.327234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.174426] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 434.902693] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 439.220611] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 446.120843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 446.212119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 446.528510] Lustre: lustre-OST0000: new disk, initializing [ 446.538080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 446.669040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.294561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.303710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.482522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.952295] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 460.711735] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.213281] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.358976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 467.700145] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 467.843374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.084679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.088287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.101996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.107340] Lustre: Skipped 2 previous similar messages [ 474.120880] Lustre: Skipped 2 previous similar messages [ 484.324937] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 484.390555] LustreError: 17959:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 484.393241] LustreError: 17959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 484.592394] Lustre: server umount lustre-OST0000 complete [ 489.440242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.452763] Lustre: Skipped 1 previous similar message [ 494.567621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 494.569820] Lustre: Skipped 7 previous similar messages [ 502.239128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 502.383640] LustreError: 18161:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 502.486694] LustreError: 18161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 502.708633] Lustre: server umount lustre-MDT0000 complete [ 504.802582] LustreError: 16308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.812955] LustreError: 16308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 506.330973] LustreError: 15008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332401 with bad export cookie 8805660048754983492 [ 506.341063] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.407746] LustreError: 15008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 506.761193] Lustre: server umount lustre-MDT0001 complete [ 510.100534] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:33:24 (1763332404) [ 512.413778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 514.137838] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:33:27 (1763332407) [ 520.176096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.553552] LustreError: 19035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.614328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.744567] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 525.796144] LustreError: 19035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.915668] LustreError: 19034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.250903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 535.101144] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 539.905797] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 543.216622] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 549.755862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 555.150306] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 559.493154] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 577.003318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.006470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 577.019856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 577.022321] LustreError: Skipped 1 previous similar message [ 577.032360] Lustre: Skipped 3 previous similar messages [ 581.768442] LustreError: 21301:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 581.784123] LustreError: 21301:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 581.917572] Lustre: server umount lustre-OST0000 complete [ 587.232903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.241822] Lustre: Skipped 2 previous similar messages [ 591.213225] Lustre: server umount lustre-MDT0000 complete [ 592.354270] LustreError: 19030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.378737] LustreError: 19030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 594.743308] LustreError: 19016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332490 with bad export cookie 8805660048754984374 [ 594.748000] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.763737] LustreError: 19016:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.098854] Lustre: server umount lustre-MDT0001 complete [ 604.488428] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 606.724240] Key type lgssc unregistered [ 606.978390] LNet: 22238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.986327] LNetError: 22238:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.008399] LNet: Removed LNI 192.168.202.119@tcp [ 607.621149] Key type .llcrypt unregistered [ 607.622699] Key type ._llcrypt unregistered [ 623.223684] Key type ._llcrypt registered [ 623.226928] Key type .llcrypt registered [ 623.317798] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 623.892120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 623.901938] alg: No test for adler32 (adler32-zlib) [ 624.910647] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 625.102869] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 626.759561] Key type lgssc registered [ 627.656242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.796473] loop: module loaded [ 633.870741] loop0: detected capacity change from 0 to 204800000 [ 633.967209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.976598] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.163341] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.192096] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 637.799887] loop0: detected capacity change from 0 to 204800000 [ 637.933853] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.947529] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.968743] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.071796] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 641.871847] loop0: detected capacity change from 0 to 204800000 [ 646.964414] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 647.024668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 647.065756] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.380432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 648.429674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 648.566648] Lustre: lustre-MDT0000: new disk, initializing [ 648.667519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.693898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 652.525747] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 661.204389] loop1: detected capacity change from 0 to 204800000 [ 666.035465] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 666.106704] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 666.310654] Lustre: lustre-OST07c6: new disk, initializing [ 666.332288] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 666.334746] Lustre: Skipped 1 previous similar message [ 666.404223] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 671.212542] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 675.416194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 675.421433] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 675.460839] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 676.736168] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 676.740143] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 676.743033] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 676.748715] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 676.753927] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 676.758148] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 694.741634] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 694.762374] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 694.766539] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.777632] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 694.795162] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 694.803611] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.152852] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 699.157727] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.161403] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 699.165564] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.169848] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.173909] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.177378] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 699.182127] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.185863] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.190523] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.195525] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.200057] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.319360] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 702.324099] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 702.335765] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 702.341183] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.347751] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.354710] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.361488] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 702.365503] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.375023] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.380113] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.386115] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.390588] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.433205] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 706.437516] Lustre: 24375:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 706.440371] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 706.444214] Lustre: 24375:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.447667] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.453923] Lustre: 24375:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.458118] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 706.460667] Lustre: 24375:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.463785] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 706.467359] Lustre: 24375:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.471643] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.474365] Lustre: 24375:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.662074] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 716.669000] Lustre: 24376:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 716.674089] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 716.677415] Lustre: 24376:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.682616] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.687144] Lustre: 24376:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.692659] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 716.701634] Lustre: 24376:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.705222] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.710833] Lustre: 24376:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.715461] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.720521] Lustre: 24376:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 767.455958] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 767.460404] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.481190] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 768.515411] LustreError: 26566:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 768.646930] Lustre: server umount lustre-OST07c6 complete [ 776.953863] LustreError: 27067:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 776.960606] LustreError: 27067:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 777.259684] Lustre: server umount lustre-MDT0000 complete [ 792.292310] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 795.082286] Key type lgssc unregistered [ 795.359119] LNet: 27900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.369317] LNetError: 27900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.385408] LNet: Removed LNI 192.168.202.119@tcp [ 796.194099] Key type .llcrypt unregistered [ 796.195894] Key type ._llcrypt unregistered [ 810.275861] Key type ._llcrypt registered [ 810.278145] Key type .llcrypt registered [ 810.373769] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 811.470755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.547557] alg: No test for adler32 (adler32-zlib) [ 812.782891] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 813.158040] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 814.935205] Key type lgssc registered [ 816.525959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 826.956985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 826.986863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.904345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.826869] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 839.144383] LustreError: 29132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 839.161511] LustreError: 29132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 842.531236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 842.942563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 846.626324] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 852.324747] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 856.473679] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 863.529619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 864.075644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.949380] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 874.302410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 875.121651] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 888.802566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.823939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.860131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.829232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.833872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.847572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.861135] Lustre: Skipped 1 previous similar message [ 894.719112] LustreError: 31580:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 894.898371] Lustre: server umount lustre-OST0000 complete [ 899.042427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 899.047220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.078286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.488194] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 904.491734] LustreError: 31781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 904.547087] Lustre: server umount lustre-MDT0000 complete [ 905.186221] LustreError: 29127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.199933] LustreError: 29127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 907.803505] LustreError: 29111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332803 with bad export cookie 12863162827627660361 [ 907.806997] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 907.812838] LustreError: 29111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 907.960826] LustreError: 31987:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 907.971616] LustreError: 31987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 908.130146] Lustre: server umount lustre-MDT0001 complete [ 919.328525] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 921.923834] Key type lgssc unregistered [ 922.211852] LNet: 32519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.218524] LNetError: 32519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.237685] LNet: Removed LNI 192.168.202.119@tcp [ 924.183405] Key type .llcrypt unregistered [ 924.184972] Key type ._llcrypt unregistered [ 941.853350] Key type ._llcrypt registered [ 941.866357] Key type .llcrypt registered [ 942.009939] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 943.412887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 943.504774] alg: No test for adler32 (adler32-zlib) [ 944.617497] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 944.944428] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 946.623705] Key type lgssc registered [ 947.891045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 956.007381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 956.031478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.597671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.989406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 968.162825] LustreError: 33770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.185187] LustreError: 33770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 972.059131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 972.456530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.078399] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 983.000814] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 986.919193] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 994.521496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 995.142352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.245438] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1003.399414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1006.448415] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1017.824037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.837969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.866925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.854519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.856435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.858378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.832812] LustreError: 36021:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1022.019133] Lustre: server umount lustre-OST0000 complete [ 1028.579368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1028.593719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.619140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.621899] Lustre: Skipped 1 previous similar message [ 1031.735740] LustreError: 36223:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1031.756854] LustreError: 36223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1031.905843] Lustre: server umount lustre-MDT0000 complete [ 1034.215662] LustreError: 33771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.239289] LustreError: 33771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1035.183504] LustreError: 33750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332930 with bad export cookie 2594785815956411172 [ 1035.189734] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.331385] LustreError: 36424:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1035.335700] LustreError: 36424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1035.615358] Lustre: server umount lustre-MDT0001 complete [ 1046.931460] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1049.679889] Key type lgssc unregistered [ 1050.016314] LNet: 36957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.029868] LNetError: 36957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.041950] LNet: Removed LNI 192.168.202.119@tcp [ 1050.852124] Key type .llcrypt unregistered [ 1050.854955] Key type ._llcrypt unregistered [ 1074.357282] Key type ._llcrypt registered [ 1074.359717] Key type .llcrypt registered [ 1074.559688] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1075.726502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.759022] alg: No test for adler32 (adler32-zlib) [ 1076.908155] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1077.258032] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1078.970646] Key type lgssc registered [ 1080.286893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.989747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.026756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.635785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.267752] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1100.779057] LustreError: 38354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.786189] LustreError: 38354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1102.705494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1103.125887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1106.477690] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1111.161904] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1114.503520] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1122.044830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1122.418633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.360592] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1134.353980] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1135.722350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1141.218215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.219840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.227531] Lustre: Skipped 2 previous similar messages [ 1141.231939] Lustre: Skipped 2 previous similar messages [ 1142.232201] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1142.349935] LustreError: 40599:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1142.402750] Lustre: server umount lustre-MDT0000 complete [ 1144.647444] LustreError: 38335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333040 with bad export cookie 15247969768751904076 [ 1144.654416] LustreError: 38335:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1144.656249] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.337337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.337717] LustreError: 39419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.344776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1149.345207] LustreError: 39419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.349081] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1149.373556] LustreError: 39419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1149.395621] Lustre: Skipped 1 previous similar message [ 1150.813079] LustreError: 40800:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1150.826749] LustreError: 40800:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1150.929639] Lustre: server umount lustre-MDT0001 complete [ 1162.575275] LustreError: 40801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.19@tcp arrived at 1763333058 with bad export cookie 15247969768751904741 [ 1162.586992] LustreError: 40801:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1179.103286] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1179.165162] LustreError: 41023:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1179.178512] LustreError: 41023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1179.394583] Lustre: server umount lustre-OST0000 complete [ 1193.112358] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1195.420390] Key type lgssc unregistered [ 1195.706846] LNet: 41862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.724759] LNetError: 41862:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.742353] LNet: Removed LNI 192.168.202.119@tcp [ 1196.457136] Key type .llcrypt unregistered [ 1196.458863] Key type ._llcrypt unregistered [ 1217.342706] Key type ._llcrypt registered [ 1217.344267] Key type .llcrypt registered [ 1217.515060] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1218.433357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1218.472384] alg: No test for adler32 (adler32-zlib) [ 1219.520133] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1219.732240] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1221.407190] Key type lgssc registered [ 1222.549720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.184455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.205086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.576878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.697370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1240.033058] LustreError: 43259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.045800] LustreError: 43259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1240.542576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1240.848581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.974157] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1248.672405] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1251.772880] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1256.477881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1256.723714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1261.444657] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1266.281130] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1275.187545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1283.556783] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1285.604139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1285.610840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.613128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1288.644271] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1288.651888] Lustre: Skipped 1 previous similar message [ 1288.702404] LustreError: 45528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1288.780353] Lustre: server umount lustre-OST0000 complete [ 1305.037475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.042368] Lustre: Skipped 1 previous similar message [ 1305.046631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.050225] Lustre: Skipped 1 previous similar message [ 1311.203139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.215368] Lustre: Skipped 3 previous similar messages [ 1319.391151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1319.463321] LustreError: 45904:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1319.477747] LustreError: 45904:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1319.621168] Lustre: server umount lustre-MDT0000 complete [ 1321.444274] LustreError: 43256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.471752] LustreError: 43256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1322.855872] LustreError: 43239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333218 with bad export cookie 625996771541528006 [ 1322.858956] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.864496] LustreError: 43239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1337.311144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1337.371414] LustreError: 46107:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1337.373483] LustreError: 46107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1337.450725] Lustre: server umount lustre-MDT0001 complete [ 1343.080753] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1345.033443] Key type lgssc unregistered [ 1345.287871] LNet: 46640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.296865] LNetError: 46640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.308781] LNet: Removed LNI 192.168.202.119@tcp [ 1345.973185] Key type .llcrypt unregistered [ 1345.975559] Key type ._llcrypt unregistered [ 1362.611911] Key type ._llcrypt registered [ 1362.613199] Key type .llcrypt registered [ 1362.736549] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1363.318400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1363.398578] alg: No test for adler32 (adler32-zlib) [ 1364.456638] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1364.622967] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1366.248087] Key type lgssc registered [ 1367.140989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1372.649329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1372.679575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.992964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.612339] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1382.653867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1383.030157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.714511] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1389.901395] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1392.839129] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1398.003116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1398.279748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.350216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1402.338489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1406.860606] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1412.530150] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1415.460452] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1427.667262] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1617h48m40s [ 1429.282767] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1434.081650] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.083888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1434.085866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.093774] Lustre: Skipped 2 previous similar messages [ 1435.090176] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1435.093082] Lustre: Skipped 2 previous similar messages [ 1436.157946] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1436.162975] Lustre: Skipped 3 previous similar messages [ 1441.216802] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 1445.344539] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.457036] LustreError: 50581:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1445.507802] Lustre: server umount lustre-MDT0000 complete [ 1447.363829] LustreError: 50378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.371747] LustreError: 50378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1448.076618] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1453.260412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.386908] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1453.516035] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (not set up) [ 1453.519239] Lustre: Skipped 3 previous similar messages [ 1453.660758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.724048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1454.706604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1456.843637] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1459.547642] Lustre: DEBUG MARKER: Wait for df (36027) ... [ 1460.909967] Lustre: DEBUG MARKER: done [ 1464.802844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1464.812783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1464.836788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.494858] LustreError: 51637:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1470.499163] LustreError: 51637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1470.584518] Lustre: server umount lustre-OST0000 complete [ 1475.553158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.556871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1475.561894] Lustre: Skipped 1 previous similar message [ 1481.702064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.704286] Lustre: Skipped 9 previous similar messages [ 1488.351116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1488.429855] LustreError: 51839:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1488.434097] LustreError: 51839:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1488.513138] Lustre: server umount lustre-MDT0000 complete [ 1491.684353] LustreError: 50310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333387 with bad export cookie 10566735769221441883 [ 1491.686899] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.690720] LustreError: 50310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1491.789702] LustreError: 52041:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1491.792754] LustreError: 52041:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1491.925880] Lustre: server umount lustre-MDT0001 complete [ 1500.334859] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1502.775893] Key type lgssc unregistered [ 1502.988115] LNet: 52574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.994377] LNetError: 52574:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.011275] LNet: Removed LNI 192.168.202.119@tcp [ 1503.709310] Key type .llcrypt unregistered [ 1503.712473] Key type ._llcrypt unregistered [ 1537.018485] Key type ._llcrypt registered [ 1537.020811] Key type .llcrypt registered [ 1537.125847] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1551.259759] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1551.940883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.061688] alg: No test for adler32 (adler32-zlib) [ 1553.067249] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1553.280310] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1554.954463] Key type lgssc registered [ 1556.260659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1564.905144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1571.482863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1578.087326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1584.387720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.920070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1591.994345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1592.023160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.399036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1593.434411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1593.534623] Lustre: lustre-MDT0000: new disk, initializing [ 1593.671888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.704170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1597.460721] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1606.999544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1607.053831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1607.165730] Lustre: 56526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1607.237298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1607.241846] Lustre: Skipped 1 previous similar message [ 1607.300359] Lustre: lustre-MDT0001: new disk, initializing [ 1607.392966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1607.427478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1607.436366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1610.917884] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1617.775323] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1620.895931] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1626.619189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1626.689804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1626.888870] Lustre: lustre-OST0000: new disk, initializing [ 1626.897404] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1627.015192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.773307] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1635.966836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1635.977820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1636.095045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1638.969511] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1643.135468] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1643.335328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1646.995632] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1647.216314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1651.680550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1651.688744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.697050] LustreError: Skipped 1 previous similar message [ 1651.707202] Lustre: Skipped 1 previous similar message [ 1651.711029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1655.036209] LustreError: 59464:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1655.186195] Lustre: server umount lustre-OST0000 complete [ 1658.848996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1658.867970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.888558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.891330] Lustre: Skipped 1 previous similar message [ 1661.923349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.926484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.610857] LustreError: 59665:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1664.615955] LustreError: 59665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1664.700754] Lustre: server umount lustre-MDT0000 complete [ 1667.636905] LustreError: 56516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333563 with bad export cookie 339769572294423520 [ 1667.643453] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1667.752886] LustreError: 59868:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1667.757829] LustreError: 59868:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1667.858093] Lustre: server umount lustre-MDT0001 complete [ 1676.241344] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 17:52:49 (1763333569) [ 1684.346740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.856245] LustreError: 60539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.877618] LustreError: 60539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1684.975633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1689.687645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1690.086513] LustreError: 60539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.208789] LustreError: 60539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1699.232996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1699.554988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1703.352794] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1709.044673] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1713.026854] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1719.315402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1719.602919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.208104] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1728.181075] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1737.479283] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1617h53m51s [ 1738.765649] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1745.018701] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1747.687252] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1747.769160] LustreError: 61689:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1767.895357] Lustre: lustre-MDT0000: Client 8c6640f5-5364-48d2-ac6b-35fbb3bc573f (at 192.168.202.19@tcp) reconnecting [ 1787.791548] LustreError: 61689:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1789.231946] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1790.907225] Lustre: DEBUG MARKER: done [ 1795.552557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.575268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.587031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.595191] Lustre: Skipped 1 previous similar message [ 1800.635363] LustreError: 63546:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1800.641297] LustreError: 63546:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1800.721070] Lustre: server umount lustre-OST0000 complete [ 1806.817893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.819823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.838167] Lustre: Skipped 2 previous similar messages [ 1806.846086] Lustre: Skipped 2 previous similar messages [ 1809.713376] LustreError: 63749:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1809.741942] LustreError: 63749:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1809.819082] Lustre: server umount lustre-MDT0000 complete [ 1813.417985] LustreError: 60521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333708 with bad export cookie 339769572294424402 [ 1813.428886] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.430905] LustreError: 60521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1813.754893] Lustre: server umount lustre-MDT0001 complete [ 1822.723058] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 1825.021403] Key type lgssc unregistered [ 1825.308598] LNet: 64482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.311388] LNetError: 64482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.331718] LNet: Removed LNI 192.168.202.119@tcp [ 1826.027883] Key type .llcrypt unregistered [ 1826.029079] Key type ._llcrypt unregistered [ 1856.403287] Key type ._llcrypt registered [ 1856.416872] Key type .llcrypt registered [ 1856.537684] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 1870.566725] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 1872.026533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.231091] alg: No test for adler32 (adler32-zlib) [ 1873.312952] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1873.630398] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 1875.338051] Key type lgssc registered [ 1876.284176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.845592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1890.603262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1896.443235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1902.685922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.291342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1909.342650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1909.364376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1910.576919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1910.616246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1910.669445] Lustre: lustre-MDT0000: new disk, initializing [ 1910.748779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.766641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1913.629352] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1923.029216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1923.091583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1923.153203] Lustre: 68432:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1923.209165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1923.211409] Lustre: Skipped 1 previous similar message [ 1923.277629] Lustre: lustre-MDT0001: new disk, initializing [ 1923.364274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.421228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1923.433244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1926.334752] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1932.086197] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1935.478889] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1941.227705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1941.293105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1941.498598] Lustre: lustre-OST0000: new disk, initializing [ 1941.504254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1941.575946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.276851] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 1951.253197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1951.259279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1951.396465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1952.134590] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1955.078896] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1955.226318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1958.305551] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1958.471596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1961.447797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1961.450155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.459699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1961.462372] Lustre: Skipped 1 previous similar message [ 1966.561433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1966.564225] Lustre: Skipped 2 previous similar messages [ 1971.683743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.243716] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1974.303262] LustreError: 71370:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1974.446871] Lustre: server umount lustre-OST0000 complete [ 1979.874749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1979.890412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.915184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.918158] Lustre: Skipped 1 previous similar message [ 1981.922982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.187069] LustreError: 71573:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1984.194885] LustreError: 71573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1984.306342] Lustre: server umount lustre-MDT0000 complete [ 1987.542970] LustreError: 68422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333882 with bad export cookie 2549043646602586267 [ 1987.547402] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1987.556266] LustreError: 68422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1987.690896] LustreError: 71774:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1987.696432] LustreError: 71774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1987.855395] Lustre: server umount lustre-MDT0001 complete [ 1995.403220] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 17:58:09 (1763333889) [ 1996.815906] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1998.411170] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 17:58:12 (1763333892) [ 2008.359995] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2011.035722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.395731] LustreError: 72824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.428422] LustreError: 72824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2011.445475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.191879] Lustre: Failing over lustre-MDT0000 [ 2012.195411] LustreError: 72852:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2012.200411] LustreError: 72852:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2012.204766] LustreError: 72852:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 2012.295629] LustreError: 72903:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2012.298719] LustreError: 72903:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2012.390888] Lustre: server umount lustre-MDT0000 complete [ 2017.575596] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:58:31 (1763333911) [ 2023.887619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.254111] LustreError: 73470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.333987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.490067] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2029.538246] LustreError: 73469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.634442] LustreError: 73469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.900181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2035.254879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.653807] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2043.449625] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2046.946966] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2053.859973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2054.118120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.675855] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2065.521742] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2075.080201] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2080.457054] LustreError: 75817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2080.460666] LustreError: 75817:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2080.569450] Lustre: server umount lustre-MDT0000 complete [ 2081.251308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2081.256649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.272380] LustreError: 73470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.417962] LustreError: 75091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333978 with bad export cookie 2549043646602587415 [ 2083.424525] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2083.428127] LustreError: 75091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2083.678406] Lustre: server umount lustre-MDT0001 complete [ 2086.422244] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2089.761147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2097.428911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.215234] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763333980/real 1763333980] req@000000002ff5454f x1848989470232576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763333996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2101.215649] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.243573] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2101.278011] Lustre: Skipped 1 previous similar message [ 2105.311204] Lustre: 66644:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763333984/real 1763333984] req@0000000007b4dc17 x1848989470232832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763334000 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.335373] Lustre: 66644:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2108.745542] LustreError: 75077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.823108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.888392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2109.791335] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763333988/real 1763333988] req@000000007b409846 x1848989470233088/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763334004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.894905] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2112.657904] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2113.892043] Lustre: 66643:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763333993/real 1763333993] req@0000000045a04dd1 x1848989470233344/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763334009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2113.918828] Lustre: 66643:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2120.126398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2124.140333] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2125.804259] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2128.962098] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2131.967972] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2139.507247] LustreError: 78277:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2139.515463] LustreError: 78277:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2139.604484] Lustre: server umount lustre-MDT0000 complete [ 2141.167308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2141.167949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.174261] LustreError: 77923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.174274] LustreError: 77923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2141.226763] Lustre: Skipped 2 previous similar messages [ 2143.113704] LustreError: 73452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334038 with bad export cookie 2549043646602589326 [ 2143.117160] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2143.120380] LustreError: 73452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2143.452208] Lustre: server umount lustre-MDT0001 complete [ 2148.779499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2157.237549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.639924] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763334041/real 1763334041] req@00000000ed6cb85d x1848989470256896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763334057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2161.685472] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2161.692071] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.848119] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x236005b0f0b6b9ea [ 2167.863923] Lustre: MGC192.168.202.119@tcp: Connection restored to 0@lo (at 0@lo) [ 2168.366366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2172.270496] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2174.883893] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763334054/real 1763334054] req@00000000e6b04bb7 x1848989470257536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763334070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2174.901891] Lustre: 66642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2177.514392] LustreError: 79348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.530865] LustreError: 79348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2180.607106] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2181.654193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2182.076039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2182.081823] Lustre: Skipped 2 previous similar messages [ 2185.869691] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2187.244670] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2190.679350] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2194.308796] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2202.599500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2202.602359] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.611638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2202.613719] Lustre: Skipped 2 previous similar messages [ 2202.624179] Lustre: Skipped 4 previous similar messages [ 2207.836652] LustreError: 80746:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2207.846664] LustreError: 80746:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2207.905370] Lustre: server umount lustre-MDT0000 complete [ 2211.489288] LustreError: 75091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334106 with bad export cookie 2549043646602590698 [ 2211.490534] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2211.501263] LustreError: 75091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2211.844119] Lustre: server umount lustre-MDT0001 complete [ 2218.074915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2220.832234] Lustre: DEBUG MARKER: files compared the same [ 2233.581433] Lustre: server umount lustre-OST0000 complete [ 2246.573442] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2248.624591] Key type lgssc unregistered [ 2248.841069] LNet: 82382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.847495] LNetError: 82382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.860982] LNet: Removed LNI 192.168.202.119@tcp [ 2249.440888] Key type .llcrypt unregistered [ 2249.442997] Key type ._llcrypt unregistered [ 2271.974058] Key type ._llcrypt registered [ 2271.976293] Key type .llcrypt registered [ 2272.081132] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2273.977240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.986133] alg: No test for adler32 (adler32-zlib) [ 2275.091589] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2275.362876] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2277.103212] Key type lgssc registered [ 2278.214208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2285.172351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2285.198063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.542590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.980173] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 2295.778204] LustreError: 83778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.816037] LustreError: 83778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2297.101448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2297.411806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2300.558171] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 2305.345819] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2308.125461] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2314.321191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2314.619057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.849644] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug +malloc all [ 2323.878553] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.986447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2341.345335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.351609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.381353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.358747] LustreError: 86048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2346.449716] Lustre: server umount lustre-OST0000 complete [ 2351.586310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.588918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.591913] Lustre: Skipped 2 previous similar messages [ 2351.597899] Lustre: Skipped 2 previous similar messages [ 2355.687921] LustreError: 86249:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2355.693124] LustreError: 86249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2355.789833] Lustre: server umount lustre-MDT0000 complete [ 2356.703858] LustreError: 83773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.730939] LustreError: 83773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2359.059504] LustreError: 83759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334254 with bad export cookie 14381914599079364800 [ 2359.063361] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.090041] LustreError: 83759:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2359.272285] LustreError: 86450:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2359.292564] LustreError: 86450:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2359.494144] Lustre: server umount lustre-MDT0001 complete [ 2368.925596] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2371.935523] Key type lgssc unregistered [ 2372.182577] LNet: 86982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.201503] LNetError: 86982:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.234715] LNet: Removed LNI 192.168.202.119@tcp [ 2373.144736] Key type .llcrypt unregistered [ 2373.152034] Key type ._llcrypt unregistered [ 2398.320366] Key type ._llcrypt registered [ 2398.327332] Key type .llcrypt registered [ 2398.480893] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2399.463311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2399.508416] alg: No test for adler32 (adler32-zlib) [ 2400.699425] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2400.896039] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2402.559180] Key type lgssc registered [ 2403.874182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2411.280554] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2411.299338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2426.975365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2432.097078] LustreError: 88353:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [ 2458.783203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.205072] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2468.963702] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2475.211879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.298652] Lustre: *** cfs_fail_loc=706, val=0*** [ 2475.595994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.689955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2479.040726] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2480.613161] LustreError: 88380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2480.638940] LustreError: 88380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2485.733932] LustreError: 88380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.028749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2486.306623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2489.672556] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2494.801720] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2497.588248] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2501.088416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.091311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.110466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.600109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.609657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.624952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.650290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2503.653013] Lustre: Skipped 1 previous similar message [ 2505.537656] LustreError: 90678:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2505.632543] Lustre: server umount lustre-OST0000 complete [ 2511.842267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2511.854448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.873978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.894092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.206574] LustreError: 90881:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2515.209847] LustreError: 90881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2515.268672] Lustre: server umount lustre-MDT0000 complete [ 2519.009582] LustreError: 90447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2519.045345] LustreError: 90447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2519.109535] LustreError: 88361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334414 with bad export cookie 11228385195102129214 [ 2519.141706] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.270461] LustreError: 91083:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2519.300176] LustreError: 91083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2519.503415] Lustre: server umount lustre-MDT0001 complete [ 2528.585776] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2530.690414] Key type lgssc unregistered [ 2530.888505] LNet: 91614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.892237] LNetError: 91614:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.909420] LNet: Removed LNI 192.168.202.119@tcp [ 2531.478186] Key type .llcrypt unregistered [ 2531.480128] Key type ._llcrypt unregistered [ 2552.295613] Key type ._llcrypt registered [ 2552.297465] Key type .llcrypt registered [ 2552.382968] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2553.146291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2553.165673] alg: No test for adler32 (adler32-zlib) [ 2554.130461] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2554.305114] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2555.983242] Key type lgssc registered [ 2556.841404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2562.183803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.200264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.531805] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2569.914338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2570.205818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.458656] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2575.328912] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.343660] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2577.837663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2578.034201] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.042828] LustreError: 93679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2578.056137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.132933] LustreError: 93684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.280214] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2585.568508] LustreError: 93684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.574952] LustreError: 93684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2587.128408] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.134975] Lustre: MGS: Client d6a7107c-cd33-4a6d-9fc2-93946deb39b3 (at 0@lo) reconnecting [ 2587.305220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2587.337266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2590.795822] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2602.979709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.983590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2603.002761] Lustre: Skipped 1 previous similar message [ 2608.105705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.110332] Lustre: Skipped 2 previous similar messages [ 2608.889404] LustreError: 95641:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2609.044738] Lustre: server umount lustre-OST0000 complete [ 2613.218279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2613.219412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.228095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.241674] Lustre: Skipped 1 previous similar message [ 2618.229916] LustreError: 95842:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2618.236300] LustreError: 95842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2618.254878] Lustre: server umount lustre-MDT0000 complete [ 2618.349100] LustreError: 93684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.353278] LustreError: 93684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2622.045822] LustreError: 96045:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2622.052663] LustreError: 96045:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2622.165820] Lustre: server umount lustre-MDT0001 complete [ 2625.781278] LustreError: 96246:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2625.793572] LustreError: 96246:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2625.906687] Lustre: server umount MGS complete [ 2637.916690] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:08:51 (1763334531) [ 2665.417532] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 2673.014966] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2680.718949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2686.022434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2690.836848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2696.094871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.536551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.596698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.772897] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2712.956319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2713.002402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2713.203599] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2713.259976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2713.327806] Lustre: lustre-MDT0001: new disk, initializing [ 2713.427231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2713.456198] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2713.467614] Lustre: 100764:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2713.480467] LustreError: 100764:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2716.602726] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2725.217817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.299284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2725.590545] Lustre: lustre-OST0000: new disk, initializing [ 2725.598650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2725.688997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.695987] Lustre: 101697:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000db0e445d x1848990183232640/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763334641 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2730.556196] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2737.479955] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2743.099257] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.107788] Lustre: MGS: Client 0651d50b-f60c-4e9a-a061-6031b8bc2af5 (at 0@lo) reconnecting [ 2743.152785] Lustre: 100049:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2743.163109] LustreError: 100049:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2743.201485] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2743.308749] Lustre: lustre-MDT0000: new disk, initializing [ 2743.479708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.514461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2743.600338] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2744.544653] Lustre: 102478:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000043069bc5 x1848990183239040/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763334655 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2746.740993] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2748.586461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2748.596625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2749.679108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2749.690227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2749.723928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2769.377976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.386260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2769.400243] Lustre: Skipped 1 previous similar message [ 2769.404190] Lustre: Skipped 1 previous similar message [ 2774.503587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2774.506336] Lustre: Skipped 1 previous similar message [ 2775.424232] LustreError: 102997:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2775.426821] LustreError: 102997:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2775.598325] Lustre: server umount lustre-OST0000 complete [ 2779.617076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2779.617136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.739911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.746301] Lustre: Skipped 3 previous similar messages [ 2785.665855] LustreError: 103199:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2785.672562] LustreError: 103199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2785.710844] Lustre: server umount lustre-MDT0000 complete [ 2795.576398] Lustre: server umount lustre-MDT0001 complete [ 2798.176889] Lustre: server umount MGS complete [ 2806.575213] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:11:40 (1763334700) [ 2819.038682] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2820.679877] Key type lgssc unregistered [ 2820.875626] LNet: 104937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.884136] LNetError: 104937:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2821.924972] LNet: Removed LNI 192.168.202.119@tcp [ 2822.529523] Key type .llcrypt unregistered [ 2822.531238] Key type ._llcrypt unregistered [ 2834.688024] Key type ._llcrypt registered [ 2834.690346] Key type .llcrypt registered [ 2834.766316] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2835.543802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2835.604973] alg: No test for adler32 (adler32-zlib) [ 2836.592608] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2836.744259] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2838.399175] Key type lgssc registered [ 2839.289864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2844.590708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2844.602540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2845.877696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.144727] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2853.781450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2854.061373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2857.054925] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2861.031815] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2863.467163] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2868.513357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2868.790600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.826986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2872.451349] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2875.668595] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2886.117267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2886.120584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.132371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.186902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2889.188425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2889.188524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.189911] Lustre: Skipped 1 previous similar message [ 2891.448795] LustreError: 108368:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2891.509384] Lustre: server umount lustre-OST0000 complete [ 2894.305291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.321808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2894.327651] Lustre: Skipped 1 previous similar message [ 2899.424424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2899.441462] Lustre: Skipped 1 previous similar message [ 2904.545094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.549913] Lustre: Skipped 1 previous similar message [ 2908.127207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2908.186917] LustreError: 108569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2908.194455] LustreError: 108569:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2908.242839] Lustre: server umount lustre-MDT0000 complete [ 2908.646237] LustreError: 106120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.662892] LustreError: 106120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2910.795317] LustreError: 106100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334806 with bad export cookie 3128408770688079509 [ 2910.797310] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.800317] LustreError: 106100:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2910.884975] LustreError: 108774:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2910.887942] LustreError: 108774:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2910.997152] Lustre: server umount lustre-MDT0001 complete [ 2918.508370] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 2920.284965] Key type lgssc unregistered [ 2920.524194] LNet: 109307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2920.534468] LNetError: 109307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.557229] LNet: Removed LNI 192.168.202.119@tcp [ 2921.308068] Key type .llcrypt unregistered [ 2921.309986] Key type ._llcrypt unregistered [ 2933.972315] Key type ._llcrypt registered [ 2933.973679] Key type .llcrypt registered [ 2934.061737] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 2935.030557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2935.052143] alg: No test for adler32 (adler32-zlib) [ 2936.044510] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2936.272653] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 2937.943230] Key type lgssc registered [ 2938.919386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2943.702503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2943.706739] LustreError: 110350:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2943.713252] LustreError: 110351:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2943.719712] LustreError: 110350:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2943.725589] LustreError: 110350:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2943.732865] LustreError: 110350:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2943.736697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.249050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.354510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2951.556277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2954.211623] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 2957.431932] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lsmod [ 2959.554871] LustreError: 111527:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2959.561764] LustreError: 111528:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2959.568227] LustreError: 111527:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2959.571925] LustreError: 111527:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2959.578566] LustreError: 111527:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2959.588031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2959.865523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.992117] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.995665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.998496] Lustre: Skipped 2 previous similar messages [ 2966.007518] Lustre: Skipped 2 previous similar messages [ 2971.107564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.113067] Lustre: Skipped 2 previous similar messages [ 2976.224092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2976.241639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.309031] LustreError: 111714:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2976.380919] Lustre: server umount lustre-MDT0000 complete [ 2979.592220] LustreError: 110369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334875 with bad export cookie 595295651668659282 [ 2979.595402] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2979.612953] LustreError: 110369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2979.719508] LustreError: 111920:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2979.723386] LustreError: 111920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2979.899919] Lustre: server umount lustre-MDT0001 complete [ 2992.963281] LustreError: 112121:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2992.967349] LustreError: 112121:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2993.088368] Lustre: server umount lustre-OST0000 complete [ 2999.791973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.146840] LustreError: 112600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.162391] LustreError: 112600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3000.222873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.208461] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3005.411967] LustreError: 112600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3009.266870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3009.500391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.318765] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3016.816697] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3019.983752] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3025.161991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3025.473863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.963485] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3033.917824] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3044.970273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3054.560536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.566231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.576907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3054.583641] Lustre: Skipped 2 previous similar messages [ 3055.591961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.596776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.687399] LustreError: 114865:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3058.693823] LustreError: 114865:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3058.876724] Lustre: server umount lustre-OST0000 complete [ 3065.825222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.831433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3065.842466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.849527] Lustre: Skipped 3 previous similar messages [ 3068.042764] LustreError: 115066:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3068.052662] LustreError: 115066:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3068.225631] Lustre: server umount lustre-MDT0000 complete [ 3070.946669] LustreError: 112600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.959860] LustreError: 112600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3072.480628] LustreError: 112579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334967 with bad export cookie 595295651668659982 [ 3072.485419] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.488375] LustreError: 112579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3072.773319] Lustre: server umount lustre-MDT0001 complete [ 3082.585451] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3085.402217] Key type lgssc unregistered [ 3085.706573] LNet: 115802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3085.711307] LNetError: 115802:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3085.726055] LNet: Removed LNI 192.168.202.119@tcp [ 3086.462530] Key type .llcrypt unregistered [ 3086.468346] Key type ._llcrypt unregistered [ 3104.105001] Key type ._llcrypt registered [ 3104.106639] Key type .llcrypt registered [ 3104.227551] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3105.014775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3105.034261] alg: No test for adler32 (adler32-zlib) [ 3106.020198] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3106.195051] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3107.831340] Key type lgssc registered [ 3108.604788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3114.198917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3114.212666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.519865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.495788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3124.706215] LustreError: 117177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.720186] LustreError: 117177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3125.323315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3125.600145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3128.369266] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3132.911805] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3136.556273] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3142.964593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3143.417937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3148.616278] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3153.315318] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3159.942389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3168.202336] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3171.333624] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3175.394151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.399981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.423811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3175.433277] Lustre: Skipped 1 previous similar message [ 3180.519043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3181.251182] LustreError: 119544:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3181.367858] Lustre: server umount lustre-OST0000 complete [ 3185.633451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.639932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.646411] Lustre: Skipped 2 previous similar messages [ 3185.652257] Lustre: Skipped 1 previous similar message [ 3190.679292] LustreError: 119749:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3190.682623] LustreError: 119749:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3190.749427] Lustre: server umount lustre-MDT0000 complete [ 3190.754419] LustreError: 117178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.765735] LustreError: 117178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3193.771802] LustreError: 117159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335089 with bad export cookie 13963880002920197544 [ 3193.780822] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.783964] LustreError: 117159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3193.891070] LustreError: 119952:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3193.900420] LustreError: 119952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3194.051276] Lustre: server umount lustre-MDT0001 complete [ 3203.669453] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3206.309889] Key type lgssc unregistered [ 3206.638507] LNet: 120482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3206.655380] LNetError: 120482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3206.685159] LNet: Removed LNI 192.168.202.119@tcp [ 3207.337158] Key type .llcrypt unregistered [ 3207.338899] Key type ._llcrypt unregistered [ 3220.906696] Key type ._llcrypt registered [ 3220.908781] Key type .llcrypt registered [ 3221.060707] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3222.189883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3222.234485] alg: No test for adler32 (adler32-zlib) [ 3223.196854] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3223.369986] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3225.047175] Key type lgssc registered [ 3226.221126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3233.266873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3233.280888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.632591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.632502] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3244.879286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3245.030088] LustreError: 121664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.066034] LustreError: 121664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3245.288635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3247.970246] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3252.633306] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3256.174796] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3262.260893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3262.543665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.277288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3270.416501] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3279.979544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3285.476649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.476842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3285.479829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3285.494436] Lustre: Skipped 1 previous similar message [ 3285.505949] LustreError: Skipped 1 previous similar message [ 3287.353112] LustreError: 123932:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3287.472385] Lustre: server umount lustre-OST0000 complete [ 3290.595907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.603686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.605884] Lustre: Skipped 1 previous similar message [ 3290.610075] Lustre: Skipped 1 previous similar message [ 3295.714136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.722197] Lustre: Skipped 2 previous similar messages [ 3296.347981] LustreError: 124133:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3296.356121] LustreError: 124133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3296.432856] Lustre: server umount lustre-MDT0000 complete [ 3299.370815] LustreError: 121644:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335194 with bad export cookie 16835837590075090989 [ 3299.375457] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.380135] LustreError: 121644:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3299.502668] LustreError: 124334:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3299.505960] LustreError: 124334:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3299.626476] Lustre: server umount lustre-MDT0001 complete [ 3307.454980] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3309.808591] Key type lgssc unregistered [ 3310.060835] LNet: 124866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3310.066497] LNetError: 124866:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3310.075364] LNet: Removed LNI 192.168.202.119@tcp [ 3310.843152] Key type .llcrypt unregistered [ 3310.847558] Key type ._llcrypt unregistered [ 3330.151108] Key type ._llcrypt registered [ 3330.152699] Key type .llcrypt registered [ 3330.225275] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3330.856986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3330.882870] alg: No test for adler32 (adler32-zlib) [ 3331.903561] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3332.097239] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3333.776225] Key type lgssc registered [ 3334.581985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3340.044429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3340.056712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.410865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3343.893477] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3350.135764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3350.410155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3353.751683] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3357.701747] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3360.717502] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3365.671248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3365.740386] LustreError: 127841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3365.749324] LustreError: 127841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3365.932944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.525494] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3370.999302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3374.520575] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3380.132874] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3383.573711] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3383.586806] Lustre: Skipped 2 previous similar messages [ 3386.945405] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3386.958489] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3387.941970] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3387.945039] Lustre: Skipped 2 previous similar messages [ 3392.109283] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3392.113883] Lustre: Skipped 1 previous similar message [ 3395.636362] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3395.641553] Lustre: Skipped 2 previous similar messages [ 3399.154510] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.19@tcp 192.168.0.[2,10]@tcp [ 3402.531041] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.19/24@tcp in log lustre-MDT0000 [ 3402.534136] Lustre: Skipped 2 previous similar messages [ 3406.594030] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.19/24@tcp [ 3406.597358] Lustre: Skipped 1 previous similar message [ 3411.938414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3411.945877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.959845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.037753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.038050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.049040] LustreError: Skipped 4 previous similar messages [ 3416.050514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.155339] LustreError: 129802:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3417.227403] Lustre: server umount lustre-OST0000 complete [ 3421.156603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.163349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.177578] Lustre: Skipped 1 previous similar message [ 3421.183526] Lustre: Skipped 2 previous similar messages [ 3426.196669] LustreError: 130003:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3426.200690] LustreError: 130003:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3426.272601] LustreError: 127763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.281845] LustreError: 127763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3426.307139] Lustre: server umount lustre-MDT0000 complete [ 3428.848243] LustreError: 126220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335324 with bad export cookie 11352038636649176740 [ 3428.849257] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3428.857103] LustreError: 126220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3428.959255] LustreError: 130204:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3428.970194] LustreError: 130204:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3429.109553] Lustre: server umount lustre-MDT0001 complete [ 3436.438689] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3438.401295] Key type lgssc unregistered [ 3438.622099] LNet: 130736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3438.629221] LNetError: 130736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3438.645109] LNet: Removed LNI 192.168.202.119@tcp [ 3439.299468] Key type .llcrypt unregistered [ 3439.302890] Key type ._llcrypt unregistered [ 3460.796339] Key type ._llcrypt registered [ 3460.798167] Key type .llcrypt registered [ 3460.907597] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3461.940600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3461.979443] alg: No test for adler32 (adler32-zlib) [ 3462.992417] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3463.227069] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3464.871129] Key type lgssc registered [ 3465.839521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.142471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.167164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.526550] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3473.547114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.549710] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.19/24@tcp [ 3476.295140] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3482.209856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3482.450506] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3482.464956] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.19/24@tcp [ 3482.504790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3485.336724] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3489.316615] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3491.920638] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3496.962431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3497.317596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.359963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3502.057945] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3506.482975] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3513.828583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3513.840553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.856636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.921631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3517.926557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.927396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.946764] Lustre: Skipped 1 previous similar message [ 3519.290095] LustreError: 134565:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3519.354172] Lustre: server umount lustre-OST0000 complete [ 3523.042088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.044727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3523.046693] Lustre: Skipped 1 previous similar message [ 3528.162578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3528.165134] Lustre: Skipped 2 previous similar messages [ 3533.281446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.284704] Lustre: Skipped 1 previous similar message [ 3536.351155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3536.418356] LustreError: 134766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3536.421014] LustreError: 134766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3536.476425] Lustre: server umount lustre-MDT0000 complete [ 3538.406117] LustreError: 132261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.421318] LustreError: 132261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3539.446067] LustreError: 132241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335434 with bad export cookie 9062264706893039296 [ 3539.457256] LustreError: 132241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3539.459535] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3539.579985] LustreError: 134969:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3539.583662] LustreError: 134969:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3539.706427] Lustre: server umount lustre-MDT0001 complete [ 3548.318489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3550.629204] Key type lgssc unregistered [ 3550.882220] LNet: 135499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3550.889844] LNetError: 135499:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3550.901699] LNet: Removed LNI 192.168.202.119@tcp [ 3551.648077] Key type .llcrypt unregistered [ 3551.653145] Key type ._llcrypt unregistered [ 3573.483204] Key type ._llcrypt registered [ 3573.484833] Key type .llcrypt registered [ 3573.605907] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3586.488471] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3587.440827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3587.456498] alg: No test for adler32 (adler32-zlib) [ 3588.562875] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3588.758763] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3590.383169] Key type lgssc registered [ 3591.301178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3599.229852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3604.718656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3610.320896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3615.714493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3621.299982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3621.346287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3621.355306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.538092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3622.576709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3622.678978] Lustre: lustre-MDT0000: new disk, initializing [ 3622.778253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.793566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3625.955589] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3634.146834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3634.226395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3634.304482] Lustre: 138866:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3634.354934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3634.358147] Lustre: Skipped 1 previous similar message [ 3634.425930] Lustre: lustre-MDT0001: new disk, initializing [ 3634.555988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3634.594460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3634.600898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3637.891080] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3644.087927] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3647.045690] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3653.376646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3653.444138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3653.640179] Lustre: lustre-OST0000: new disk, initializing [ 3653.645458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3653.720496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3658.636238] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3663.464135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3663.488601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3663.632591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3666.307944] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3670.385713] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3670.657302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3674.548036] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3674.739628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3679.201277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3679.205526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.219269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3682.445176] LustreError: 141805:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3682.579915] Lustre: server umount lustre-OST0000 complete [ 3685.867193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3685.880454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.885511] Lustre: Skipped 1 previous similar message [ 3685.889739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.893035] Lustre: Skipped 1 previous similar message [ 3689.442949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3689.448038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.561478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.565873] Lustre: Skipped 2 previous similar messages [ 3698.661501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3698.669721] Lustre: Skipped 1 previous similar message [ 3700.191155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3700.284500] LustreError: 142006:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3700.296474] LustreError: 142006:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3700.359864] Lustre: server umount lustre-MDT0000 complete [ 3703.535594] LustreError: 138857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335598 with bad export cookie 9478807712783439770 [ 3703.542784] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3703.551388] LustreError: 138857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3703.741244] LustreError: 142209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3703.747878] LustreError: 142209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3703.946533] Lustre: server umount lustre-MDT0001 complete [ 3710.863439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.242160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.141545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3716.579661] LustreError: 142685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.589101] LustreError: 142685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3720.145951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3720.412593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3723.153584] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3727.228802] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3730.067430] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3735.127585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3735.384967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3739.573915] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3743.715392] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3747.361344] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3747.536522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3750.429497] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3750.604529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3759.849936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3759.914370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3760.013170] Lustre: lustre-OST0001: new disk, initializing [ 3760.016310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3760.131094] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.735936] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3766.343521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3766.347211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3766.438291] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3771.203901] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3771.369058] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3774.621570] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3774.808625] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3777.919969] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99710868a000.ost_server_uuid 50 [ 3779.212168] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99710868a000.ost_server_uuid in FULL state after 0 sec [ 3787.241434] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3787.245431] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3787.250743] LustreError: Skipped 1 previous similar message [ 3787.265419] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3791.179239] LustreError: 146930:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3791.193151] LustreError: 146930:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3791.244313] Lustre: server umount lustre-OST0001 complete [ 3792.360786] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.369734] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3795.936306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3795.945951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.952359] Lustre: Skipped 1 previous similar message [ 3797.479726] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.488319] LustreError: 146137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3802.597815] LustreError: 144282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.610983] LustreError: 144282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3807.717031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.719721] LustreError: 144282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.729451] Lustre: Skipped 7 previous similar messages [ 3807.743850] LustreError: 144282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3808.231040] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3808.279104] LustreError: 147131:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3808.281855] LustreError: 147131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3808.464365] Lustre: server umount lustre-OST0000 complete [ 3812.835605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.837585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3812.845627] Lustre: Skipped 1 previous similar message [ 3818.368950] LustreError: 147337:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3818.376038] LustreError: 147337:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3818.450643] Lustre: server umount lustre-MDT0000 complete [ 3821.627321] LustreError: 142666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335717 with bad export cookie 9478807712783440645 [ 3821.631190] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.638945] LustreError: 142666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3821.917705] Lustre: server umount lustre-MDT0001 complete [ 3836.173151] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 3838.213658] Key type lgssc unregistered [ 3838.453563] LNet: 148527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3838.457839] LNetError: 148527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3838.468656] LNet: Removed LNI 192.168.202.119@tcp [ 3839.155489] Key type .llcrypt unregistered [ 3839.158864] Key type ._llcrypt unregistered [ 3868.474849] Key type ._llcrypt registered [ 3868.477944] Key type .llcrypt registered [ 3868.644230] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 3883.093606] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 3884.118433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.421062] alg: No test for adler32 (adler32-zlib) [ 3885.666819] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3885.886510] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 3887.567702] Key type lgssc registered [ 3888.849966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.240773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3904.347064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3909.695638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3915.145538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.744157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.807452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3920.823818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.005888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3922.038604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3922.098656] Lustre: lustre-MDT0000: new disk, initializing [ 3922.213829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.244310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3925.642788] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3935.098809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3935.168894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3935.244181] Lustre: 152478:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3935.323343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3935.327936] Lustre: Skipped 1 previous similar message [ 3935.397534] Lustre: lustre-MDT0001: new disk, initializing [ 3935.493299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.546797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3935.564470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3938.998820] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3944.644544] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3947.364442] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3952.094444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3952.196352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3952.411364] Lustre: lustre-OST0000: new disk, initializing [ 3952.416416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3952.517416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3956.559672] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 3959.401259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3959.409901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3959.527046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3962.678660] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3965.925660] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3966.118060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3969.307860] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3969.443360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3975.137670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3975.139505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3975.141102] LustreError: Skipped 1 previous similar message [ 3975.148790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3975.150773] Lustre: Skipped 1 previous similar message [ 3975.159745] Lustre: Skipped 1 previous similar message [ 3980.259229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.262635] Lustre: Skipped 1 previous similar message [ 3985.377650] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3985.381078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3985.392987] Lustre: Skipped 1 previous similar message [ 3985.428147] LustreError: 155420:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3985.542131] Lustre: server umount lustre-OST0000 complete [ 3990.497897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.508910] Lustre: Skipped 1 previous similar message [ 3990.509463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.717744] LustreError: 155623:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3994.721839] LustreError: 155623:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3994.787936] Lustre: server umount lustre-MDT0000 complete [ 3997.309802] LustreError: 152469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335892 with bad export cookie 4179997056439059158 [ 3997.311299] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3997.319322] LustreError: 152469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3997.453579] LustreError: 155824:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3997.456642] LustreError: 155824:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3997.606362] Lustre: server umount lustre-MDT0001 complete [ 4003.823554] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 18:31:37 (1763335897) [ 4014.202178] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4020.711366] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4027.220895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4031.815671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4036.254860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4040.637233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4045.948748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4046.015556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.173989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4046.208395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4046.262348] Lustre: lustre-MDT0000: new disk, initializing [ 4046.377714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.388069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4049.079469] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4057.841581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4057.965568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4058.077111] Lustre: 158814:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4058.126622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4058.129467] Lustre: Skipped 1 previous similar message [ 4058.215575] Lustre: lustre-MDT0001: new disk, initializing [ 4058.287916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.309784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4058.320585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4062.153650] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4067.713833] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4070.244474] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4076.093897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4076.138434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4076.353398] Lustre: lustre-OST0000: new disk, initializing [ 4076.359167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4076.440660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4081.128657] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4085.796079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4085.807335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4085.885378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4086.577224] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4089.883061] Lustre: Failing over lustre-OST0000 [ 4089.926476] LustreError: 161254:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4089.932418] LustreError: 161254:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4090.039375] Lustre: server umount lustre-OST0000 complete [ 4106.236500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4106.459785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.472095] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4106.704420] Lustre: 150688:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335986/real 1763335986] req@000000006f83d121 x1848991579964928/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763336002 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4106.721826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.736406] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4107.234220] Lustre: 150691:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335986/real 1763335986] req@00000000628b6123 x1848991579964800/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763336002 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4108.777247] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4108.778458] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4110.298893] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4112.289234] Lustre: 150691:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335991/real 1763335991] req@000000004fcd3cd6 x1848991579966848/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763336007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4112.313885] Lustre: 150691:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4113.042989] Lustre: Failing over lustre-MDT0000 [ 4113.141166] LustreError: 162234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4113.148937] LustreError: 162234:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4113.229143] Lustre: server umount lustre-MDT0000 complete [ 4113.873602] LustreError: 158821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.878599] LustreError: 158821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4114.917189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4114.920739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.933902] Lustre: Skipped 1 previous similar message [ 4114.940411] LustreError: 158825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.449283] LustreError: 158825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.471318] LustreError: 158825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4116.946042] Lustre: 150690:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763335996/real 1763335996] req@0000000018065266 x1848991579967872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763336012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4118.982019] LustreError: 158820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4122.064049] Lustre: 150689:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763336001/real 1763336001] req@00000000483d3e60 x1848991579969024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763336017 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4124.098159] LustreError: 158821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.119928] LustreError: 158821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4129.553581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.618719] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4129.860737] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4132.848053] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4134.335899] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4134.896296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4134.899894] Lustre: Skipped 1 previous similar message [ 4134.908777] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4134.937039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4139.489292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4139.493989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.504128] Lustre: Skipped 2 previous similar messages [ 4139.508822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4139.511917] Lustre: Skipped 1 previous similar message [ 4144.889699] LustreError: 163118:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4144.906140] LustreError: 163118:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4145.025552] Lustre: server umount lustre-OST0000 complete [ 4150.243804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4150.244293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4150.244932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4150.244936] Lustre: Skipped 1 previous similar message [ 4150.256929] Lustre: Skipped 2 previous similar messages [ 4154.414249] Lustre: server umount lustre-MDT0000 complete [ 4155.362612] LustreError: 158820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.376931] LustreError: 158820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4157.863917] LustreError: 159593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336053 with bad export cookie 4179997056439061272 [ 4157.868029] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4157.876817] LustreError: 159593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4158.226332] Lustre: server umount lustre-MDT0001 complete [ 4167.111842] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4169.435485] Key type lgssc unregistered [ 4169.702849] LNet: 164053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4169.719085] LNetError: 164053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4169.732221] LNet: Removed LNI 192.168.202.119@tcp [ 4170.478572] Key type .llcrypt unregistered [ 4170.482684] Key type ._llcrypt unregistered [ 4202.873376] Key type ._llcrypt registered [ 4202.875136] Key type .llcrypt registered [ 4202.953681] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4203.809504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4203.833155] alg: No test for adler32 (adler32-zlib) [ 4204.812725] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4204.999100] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4206.647186] Key type lgssc registered [ 4207.431755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4213.572942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4213.592905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.054471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.551223] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4224.309726] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4225.509129] LustreError: 165555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.523474] LustreError: 165555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4227.050329] LustreError: 165584:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4227.055357] LustreError: 165584:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4227.059723] LustreError: 165584:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 4227.153465] LustreError: 166107:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4227.257236] Lustre: server umount lustre-MDT0000 complete [ 4238.806994] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4245.772490] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4253.460560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4259.749391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4265.401295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4270.888480] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4276.752694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4276.808709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4276.964496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4276.994463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4277.044514] Lustre: lustre-MDT0000: new disk, initializing [ 4277.112240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.129755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4280.649306] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4290.564768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4290.641854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4290.737336] Lustre: 168752:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.822966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.826404] Lustre: Skipped 1 previous similar message [ 4290.915242] Lustre: lustre-MDT0001: new disk, initializing [ 4291.049167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.092900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4291.109506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4294.921108] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4300.819170] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4304.452318] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4311.509295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4311.598158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4311.903927] Lustre: lustre-OST0000: new disk, initializing [ 4311.911110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4312.047346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.741824] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4320.338887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4320.356691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4320.469374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4325.404513] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4329.748060] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4331.186062] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:37:05 (1763336225) [ 4552.160726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4552.175173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.209562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.256811] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.265120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.268283] Lustre: Skipped 1 previous similar message [ 4556.272610] Lustre: Skipped 1 previous similar message [ 4558.244699] LustreError: 172273:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4558.250138] LustreError: 172273:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4558.326543] Lustre: server umount lustre-MDT0000 complete [ 4561.376304] LustreError: 168759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.399136] LustreError: 168759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4561.957649] LustreError: 169531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336457 with bad export cookie 632276245635328367 [ 4561.959426] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.970175] LustreError: 169531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4566.498550] LustreError: 168758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.499811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.592922] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4568.191263] LustreError: 172477:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4568.203574] LustreError: 172477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4568.377674] Lustre: server umount lustre-MDT0001 complete [ 4578.362278] LustreError: 172679:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4578.371411] LustreError: 172679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4578.506848] Lustre: server umount lustre-OST0000 complete [ 4585.857398] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4592.914124] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4601.228228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4606.651415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.041331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4617.975857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4623.790910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4623.851650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.060304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4624.105170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4624.189567] Lustre: lustre-MDT0000: new disk, initializing [ 4624.289126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.310704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4627.335139] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4635.949777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4636.004629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4636.085705] Lustre: 175020:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4636.152369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4636.155275] Lustre: Skipped 1 previous similar message [ 4636.229091] Lustre: lustre-MDT0001: new disk, initializing [ 4636.324586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4636.383831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4636.398489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4639.513271] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4646.731787] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4649.921829] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4656.065735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4656.147507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4656.462974] Lustre: lustre-OST0000: new disk, initializing [ 4656.466128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4656.552225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4661.518933] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4664.373520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4664.380789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4664.500423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4668.155424] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4671.580172] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4671.801297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4674.974548] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4675.186555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4680.162432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.164658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4680.231508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4683.025228] LustreError: 177960:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4683.045699] LustreError: 177960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4683.229911] Lustre: server umount lustre-OST0000 complete [ 4687.840677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4687.847640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.859332] Lustre: Skipped 1 previous similar message [ 4687.864190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4687.868800] Lustre: Skipped 1 previous similar message [ 4692.891547] LustreError: 178161:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4692.895096] LustreError: 178161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4693.078232] Lustre: server umount lustre-MDT0000 complete [ 4695.520948] LustreError: 175032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.538460] LustreError: 175032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4696.638926] LustreError: 175015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336592 with bad export cookie 632276245635487197 [ 4696.640878] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4696.683030] LustreError: 175015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4697.059825] Lustre: server umount lustre-MDT0001 complete [ 4704.674107] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:43:18 (1763336598) [ 4720.588449] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4729.053591] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4737.643634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4744.957061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.797779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4758.614636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.500198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.564416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.735436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.760144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4764.839174] Lustre: lustre-MDT0000: new disk, initializing [ 4764.920177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.935166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4768.188341] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4777.394346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.437962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4777.570752] Lustre: 181354:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4777.859644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4780.647559] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4786.918315] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4789.474554] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4794.709915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4794.759281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4800.131693] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4804.682177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4804.685487] Lustre: Skipped 1 previous similar message [ 4804.688436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4804.768966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4806.653872] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4815.332651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.338774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4815.339379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4815.344292] Lustre: Skipped 2 previous similar messages [ 4815.347131] Lustre: Skipped 2 previous similar messages [ 4819.132851] LustreError: 183899:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4819.135495] LustreError: 183899:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4819.246375] Lustre: server umount lustre-OST0000 complete [ 4824.034456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4824.038065] LustreError: Skipped 1 previous similar message [ 4828.537183] Lustre: server umount lustre-MDT0000 complete [ 4830.689285] LustreError: 181365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.705630] LustreError: 181365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4832.116196] LustreError: 181345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336727 with bad export cookie 632276245635488079 [ 4832.119848] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4832.144501] LustreError: 181345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4832.447914] Lustre: server umount lustre-MDT0001 complete [ 4839.748382] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:45:33 (1763336733) [ 4851.372613] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 4859.536756] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4869.492811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.699727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4881.878207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4887.593779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4893.601534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4893.681511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.859501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4893.888577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4893.891856] Lustre: Skipped 3 previous similar messages [ 4893.949106] Lustre: lustre-MDT0000: new disk, initializing [ 4893.960764] Lustre: Skipped 2 previous similar messages [ 4894.083252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.085449] Lustre: Skipped 2 previous similar messages [ 4894.118647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4897.368761] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4905.234677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4905.273890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4905.357658] Lustre: 187294:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4905.540740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4908.268418] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4913.626113] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4916.105096] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4921.040817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.088936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4926.481159] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 4930.204375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4933.017699] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4940.768607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.774961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.785895] Lustre: Skipped 3 previous similar messages [ 4940.791127] Lustre: Skipped 4 previous similar messages [ 4944.438148] LustreError: 189840:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4944.452209] LustreError: 189840:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4944.539264] Lustre: server umount lustre-OST0000 complete [ 4953.757456] Lustre: server umount lustre-MDT0000 complete [ 4956.134494] LustreError: 187305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.151970] LustreError: 187305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4956.428489] LustreError: 187285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336851 with bad export cookie 632276245635489164 [ 4956.434277] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4956.438149] LustreError: 187285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4956.724352] Lustre: server umount lustre-MDT0001 complete [ 4964.646537] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 4967.050049] Key type lgssc unregistered [ 4967.322509] LNet: 190776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4967.337557] LNetError: 190776:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4967.356641] LNet: Removed LNI 192.168.202.119@tcp [ 4968.046520] Key type .llcrypt unregistered [ 4968.048973] Key type ._llcrypt unregistered [ 4984.877837] Key type ._llcrypt registered [ 4984.882317] Key type .llcrypt registered [ 4984.975575] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 4985.586552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4985.643366] alg: No test for adler32 (adler32-zlib) [ 4986.588936] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4986.742490] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 4988.378084] Key type lgssc registered [ 4989.232737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4994.557589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4994.577221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.912658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.361476] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5006.312456] LustreError: 192151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.344153] LustreError: 192151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5007.384547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5008.037079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.274531] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5019.464982] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5023.322742] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5028.926790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5029.179822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.972694] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5038.828676] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5041.470123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5062.116165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5062.121060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.126656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5062.130573] Lustre: Skipped 1 previous similar message [ 5062.140185] Lustre: Skipped 1 previous similar message [ 5065.993853] LustreError: 194426:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5066.190896] Lustre: server umount lustre-OST0000 complete [ 5072.353138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.358248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.372641] Lustre: Skipped 1 previous similar message [ 5075.561858] LustreError: 194627:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5075.579591] LustreError: 194627:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5075.691178] Lustre: server umount lustre-MDT0000 complete [ 5077.473032] LustreError: 192152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5077.478465] LustreError: 192152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5079.385333] LustreError: 192133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336974 with bad export cookie 7163159617240717291 [ 5079.387884] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5079.424030] LustreError: 192133:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5079.610720] LustreError: 194829:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5079.647509] LustreError: 194829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5079.863852] Lustre: server umount lustre-MDT0001 complete [ 5088.556967] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 5090.893395] Key type lgssc unregistered [ 5091.138768] LNet: 195360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5091.147941] LNetError: 195360:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5091.167365] LNet: Removed LNI 192.168.202.119@tcp [ 5092.003637] Key type .llcrypt unregistered [ 5092.010785] Key type ._llcrypt unregistered [ 5110.976191] Key type ._llcrypt registered [ 5110.978131] Key type .llcrypt registered [ 5111.052755] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5111.872373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5111.903671] alg: No test for adler32 (adler32-zlib) [ 5112.957264] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5113.195952] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 5114.905427] Key type lgssc registered [ 5115.950079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5122.035369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5122.051373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.487790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.486504] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5132.434831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5132.736092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.144881] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5138.725441] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5141.344855] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5146.342753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5146.657949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5148.717969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5151.337801] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5154.966866] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5163.116276] LustreError: 198983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5163.251200] Lustre: server umount lustre-OST0000 complete [ 5168.255929] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff997110823800.ost_server_uuid 50 [ 5180.895158] Lustre: 196047:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337059/real 1763337059] req@000000002bc2ecc9 x1848992866523520/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763337075 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5180.933130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.031218] Lustre: 196048:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337063/real 1763337063] req@00000000653b8bc7 x1848992866523776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763337079 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5184.053462] Lustre: 196048:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5184.059855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.890752] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5190.159936] Lustre: 196047:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337068/real 1763337068] req@00000000b90bd1b8 x1848992866525568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763337084 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5190.194737] Lustre: 196047:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5194.250295] Lustre: 196048:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337073/real 1763337073] req@00000000b0383248 x1848992866526720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763337089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5199.333674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.337143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5199.343781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5199.355056] Lustre: Skipped 1 previous similar message [ 5204.092929] LustreError: 199382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5204.107567] LustreError: 199382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5204.182156] Lustre: server umount lustre-MDT0000 complete [ 5204.449568] LustreError: 197898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5204.471650] LustreError: 197898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5207.294373] LustreError: 196717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337102 with bad export cookie 4189350006495485334 [ 5207.294730] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5207.298959] LustreError: 196717:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5207.432649] LustreError: 199584:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5207.441550] LustreError: 199584:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5207.609665] Lustre: server umount lustre-MDT0001 complete [ 5214.677454] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:51:48 (1763337108) [ 5220.049458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.418302] LustreError: 200256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.510730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.522743] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5225.952699] LustreError: 200256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.036992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5230.057304] LustreError: 200256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5230.456838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5233.640210] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5237.804667] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5240.682627] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5245.459269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5245.696472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.854151] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5253.827760] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5258.714768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5258.790942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5258.941461] Lustre: lustre-OST0001: new disk, initializing [ 5258.953394] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5260.489446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5260.504195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5260.605393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5263.745649] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5264.686035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5268.807436] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5274.575698] Lustre: lustre-OST0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 5274.579576] Lustre: Skipped 1 previous similar message [ 5275.109633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.114744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5276.726147] LustreError: 203462:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5276.729093] LustreError: 203462:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5276.759719] Lustre: server umount lustre-OST0000 complete [ 5279.686189] LustreError: 201853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.458764] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5280.575917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5283.306645] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5283.490694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5289.922332] LustreError: 202864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.938468] LustreError: 202864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5295.590688] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.593315] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5295.604284] Lustre: Skipped 2 previous similar messages [ 5295.612245] Lustre: Skipped 3 previous similar messages [ 5300.708285] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5301.230491] LustreError: 204185:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5301.242845] LustreError: 204185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5301.455320] Lustre: server umount lustre-OST0001 complete [ 5305.827437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.828907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5305.832590] Lustre: Skipped 1 previous similar message [ 5305.838732] Lustre: Skipped 2 previous similar messages [ 5311.156079] LustreError: 204386:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5311.159452] LustreError: 204386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5311.228032] Lustre: server umount lustre-MDT0000 complete [ 5314.515992] LustreError: 200237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337209 with bad export cookie 4189350006495486398 [ 5314.526103] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5314.530250] LustreError: 200237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5314.904629] Lustre: server umount lustre-MDT0001 complete [ 5320.061675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5323.224104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5326.706229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5330.169735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5336.635470] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:53:50 (1763337230) [ 5342.247730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.555699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5342.610927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5342.790104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.793335] Lustre: Skipped 1 previous similar message [ 5346.180381] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5353.272418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5353.463402] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5353.495622] Lustre: 206079:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5356.944694] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5361.304324] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5364.236528] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5369.526401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5369.658898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5374.974815] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5379.167142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5379.885685] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5385.709487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5385.847857] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5386.042359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5386.046026] Lustre: Skipped 2 previous similar messages [ 5391.264221] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5395.821102] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5396.101164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5399.523118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.526554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.528749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.557287] Lustre: Skipped 1 previous similar message [ 5399.560237] Lustre: Skipped 2 previous similar messages [ 5412.319225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5412.366259] LustreError: 209277:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5412.378996] LustreError: 209277:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5412.461242] Lustre: server umount lustre-OST0000 complete [ 5416.900096] LustreError: 207706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.921412] LustreError: 207706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5427.179367] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.187239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5427.188153] LustreError: Skipped 2 previous similar messages [ 5427.190415] Lustre: Skipped 11 previous similar messages [ 5432.093527] Lustre: server umount lustre-OST0001 complete [ 5435.877747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5435.883341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.890784] Lustre: Skipped 2 previous similar messages [ 5449.183148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5449.262489] LustreError: 209701:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5449.264824] LustreError: 209701:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5449.357918] Lustre: server umount lustre-MDT0000 complete [ 5451.748309] LustreError: 206090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5451.764759] LustreError: 206090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5451.880313] LustreError: 206071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337347 with bad export cookie 4189350006495487805 [ 5451.881288] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5451.889463] LustreError: 206071:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5452.110182] Lustre: server umount lustre-MDT0001 complete [ 5456.746609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5460.153546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5464.455789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5468.667065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5475.684800] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:56:09 (1763337369) [ 5487.829971] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 5495.080697] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 5503.342222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5509.171863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5513.592111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5519.040688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5524.972265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5525.013812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.244046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5525.279110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5525.357891] Lustre: lustre-MDT0000: new disk, initializing [ 5525.464190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.481967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5528.551625] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5537.669083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5537.717512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5537.817491] Lustre: 213727:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5537.880079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5537.883336] Lustre: Skipped 1 previous similar message [ 5537.948342] Lustre: lustre-MDT0001: new disk, initializing [ 5538.086854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5538.100668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5541.954452] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5548.217599] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5551.443887] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5557.784888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5557.879230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5558.142454] Lustre: lustre-OST0000: new disk, initializing [ 5558.155117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5563.993241] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5566.529917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5566.549585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5566.667376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5571.796539] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.890771] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5576.223183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5580.764250] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5580.913509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5587.424048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.426610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.432705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.438203] Lustre: Skipped 2 previous similar messages [ 5587.448366] Lustre: Skipped 8 previous similar messages [ 5597.151130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5597.186650] LustreError: 216668:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5597.188553] LustreError: 216668:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5597.352564] Lustre: server umount lustre-OST0000 complete [ 5607.349153] Lustre: server umount lustre-MDT0000 complete [ 5607.904721] LustreError: 213738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.926081] LustreError: 213738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5610.478616] LustreError: 213719:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337505 with bad export cookie 4189350006495489597 [ 5610.483516] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.773587] Lustre: server umount lustre-MDT0001 complete [ 5618.832633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.524697] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5630.311625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5634.064732] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5638.834537] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5642.135763] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5648.696819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5654.040659] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5659.228821] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5663.195386] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5665.473662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5668.885274] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5669.148166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5674.980808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5674.983843] LustreError: Skipped 1 previous similar message [ 5674.987891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5674.990484] Lustre: Skipped 4 previous similar messages [ 5685.215940] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5685.367164] Lustre: server umount lustre-OST0000 complete [ 5691.037196] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5694.983326] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5723.990206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5724.321896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.329435] Lustre: Skipped 5 previous similar messages [ 5729.513288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5734.839843] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5739.690930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.707682] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.708835] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5742.129791] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5745.120143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.123420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.133038] Lustre: Skipped 4 previous similar messages [ 5751.044221] LustreError: 221945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5751.046276] LustreError: 221945:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5751.216109] Lustre: server umount lustre-OST0000 complete [ 5761.416741] Lustre: server umount lustre-MDT0000 complete [ 5765.114363] LustreError: 217528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337660 with bad export cookie 4189350006495490479 [ 5765.128448] LustreError: 217528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5765.132586] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5765.580470] Lustre: server umount lustre-MDT0001 complete [ 5774.583206] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:01:08 (1763337668) [ 5781.423047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.776722] LustreError: 223032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.817200] LustreError: 223032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5785.771285] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5795.561374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5800.536222] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5806.627339] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5810.516599] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5817.347687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5823.826815] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5829.701768] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5833.937626] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5834.168989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5837.941082] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5838.136981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5844.526528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5844.629830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5844.763201] Lustre: lustre-OST0001: new disk, initializing [ 5844.768049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5849.996214] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5851.203652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5851.219874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5851.285551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5857.413745] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5861.442825] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5861.704662] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5865.309782] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5865.471488] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5872.096289] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5872.110188] LustreError: Skipped 1 previous similar message [ 5872.125624] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5872.130462] Lustre: Skipped 14 previous similar messages [ 5873.356819] Lustre: server umount lustre-OST0001 complete [ 5878.369599] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5878.613666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5882.262639] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5882.453590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5910.424810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5915.352908] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 5915.642975] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5915.654977] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5915.658576] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5915.668707] Lustre: Skipped 1 previous similar message [ 5920.215644] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5921.710220] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5929.458991] Lustre: server umount lustre-OST0001 complete [ 5950.599818] Lustre: server umount lustre-OST0000 complete [ 5960.495897] Lustre: server umount lustre-MDT0000 complete [ 5963.884074] LustreError: 229323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337859 with bad export cookie 4189350006495491445 [ 5963.889260] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5963.905969] LustreError: 229323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5970.313702] Lustre: server umount lustre-MDT0001 complete [ 5975.267465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5978.836904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5982.614829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5987.005890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5993.761314] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:04:47 (1763337887) [ 6000.143481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.471986] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6000.507852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6000.783833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6000.792143] Lustre: Skipped 5 previous similar messages [ 6003.943401] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6011.703943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6011.880872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6011.914716] Lustre: 231019:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6016.235502] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6022.261503] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6026.256569] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.416185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6034.546980] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6040.706380] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6046.524771] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6056.400211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6056.455990] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6062.571450] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6068.911896] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6071.204115] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6075.447393] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6075.615425] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6079.638800] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6079.813337] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6083.519700] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff997110051000.ost_server_uuid 50 [ 6085.099837] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff997110051000.ost_server_uuid in IDLE state after 0 sec [ 6085.989568] Lustre: Permanently deactivating lustre-OST0001 [ 6086.002679] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6089.340969] Lustre: Permanently reactivating lustre-OST0001 [ 6095.840507] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6095.845731] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.855309] LustreError: Skipped 6 previous similar messages [ 6095.863030] Lustre: Skipped 10 previous similar messages [ 6099.394112] LustreError: 235002:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6099.399459] LustreError: 235002:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6099.468206] Lustre: server umount lustre-OST0001 complete [ 6100.974243] LustreError: 232956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.991514] LustreError: 232956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 33 previous similar messages [ 6117.343135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6117.499180] Lustre: server umount lustre-OST0000 complete [ 6126.902460] Lustre: server umount lustre-MDT0000 complete [ 6130.310041] LustreError: 231010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338025 with bad export cookie 4189350006495492733 [ 6130.324954] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6130.326919] LustreError: 231010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6130.663368] Lustre: server umount lustre-MDT0001 complete [ 6136.095586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6139.818904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6143.494728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6146.885486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6153.381920] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:07:26 (1763338046) [ 6154.611752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6160.646369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.851732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6160.876383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6160.884758] Lustre: Skipped 5 previous similar messages [ 6164.393393] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6171.585264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6171.676818] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6171.715398] Lustre: 237169:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6175.362728] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6180.257254] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.507642] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6189.728496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6189.820945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6189.972792] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6189.991316] Lustre: Permanently deactivating lustre-OST0000 [ 6190.006817] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6194.864489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6196.766696] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6196.779939] LustreError: 239269:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 6196.823452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6199.537969] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6205.081691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6205.147114] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6210.911398] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6212.884408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6216.444424] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6218.496157] Lustre: Permanently reactivating lustre-OST0000 [ 6218.537685] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6218.540858] Lustre: Skipped 2 previous similar messages [ 6220.111713] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6220.126491] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.144648] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6220.178186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.191315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6233.584411] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6233.598933] Lustre: Skipped 24 previous similar messages [ 6235.261419] Lustre: server umount lustre-OST0001 complete [ 6244.837630] Lustre: server umount lustre-OST0000 complete [ 6250.318482] Lustre: server umount lustre-MDT0000 complete [ 6253.715855] LustreError: 237894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338149 with bad export cookie 4189350006495494385 [ 6253.721832] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6253.724547] LustreError: 237894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6254.019515] Lustre: server umount lustre-MDT0001 complete [ 6262.438662] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6265.014815] Key type lgssc unregistered [ 6265.254275] LNet: 241582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6265.263341] LNetError: 241582:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6265.280428] LNet: Removed LNI 192.168.202.119@tcp [ 6265.979265] Key type .llcrypt unregistered [ 6265.981126] Key type ._llcrypt unregistered [ 6285.197070] Key type ._llcrypt registered [ 6285.198760] Key type .llcrypt registered [ 6285.361986] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6286.314404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6286.344631] alg: No test for adler32 (adler32-zlib) [ 6287.328944] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 6287.564572] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6289.281063] Key type lgssc registered [ 6290.280099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6293.362837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6298.734449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6298.754174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6300.110181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6303.405390] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6310.375926] LustreError: 243012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.402089] LustreError: 243012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6311.860128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.949455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6312.070099] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6312.096807] Lustre: 243001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6312.103330] Lustre: Permanently deactivating lustre-MDT0001 [ 6312.119724] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6312.399390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6314.712268] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6314.716033] LustreError: 243043:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6314.724052] LustreError: 243043:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6314.740589] LustreError: 243043:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6315.482399] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6319.348394] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6321.944535] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6327.373719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6327.606515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.406712] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6336.807822] LustreError: 244629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.855387] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6337.400737] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6342.126557] LustreError: 244626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.543330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6343.756586] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6344.753075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6344.759735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6344.766272] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6348.319721] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6353.389313] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6356.355677] Lustre: Permanently reactivating lustre-MDT0001 [ 6356.374581] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6356.378990] Lustre: Skipped 2 previous similar messages [ 6361.594426] Lustre: lustre-MDT0001: Client 658b278a-c1dc-4d9c-96a5-ee7deb004fc5 (at 192.168.202.19@tcp) reconnecting [ 6364.970188] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6366.447195] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6369.852846] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6369.993957] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6373.025350] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6373.212666] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6375.405223] Lustre: Permanently deactivating lustre-MDT0001 [ 6375.421543] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6375.425942] Lustre: Skipped 2 previous similar messages [ 6383.197108] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6389.220194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6389.232643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6389.244773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.826838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.827248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.834958] Lustre: Skipped 1 previous similar message [ 6394.988528] LustreError: 247081:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6395.077652] Lustre: server umount lustre-MDT0000 complete [ 6398.165594] LustreError: 246119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338293 with bad export cookie 2579779564246600092 [ 6398.176371] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.287690] LustreError: 247283:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6398.289745] LustreError: 247283:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6398.421938] Lustre: server umount lustre-MDT0001 complete [ 6407.485467] LustreError: 247483:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6407.492541] LustreError: 247483:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6407.535816] Lustre: server umount lustre-OST0000 complete [ 6417.230901] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:11:51 (1763338311) [ 6427.223880] LustreError: 248489:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6427.226702] LustreError: 248489:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6427.416443] Lustre: server umount lustre-OST0001 complete [ 6432.466545] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6439.405254] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6446.579487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6451.406676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6457.062098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6462.935477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6468.972704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6469.039891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.268985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6469.278628] Lustre: Skipped 2 previous similar messages [ 6469.321313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6469.422665] Lustre: lustre-MDT0000: new disk, initializing [ 6469.485942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.498121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6472.742546] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6482.170153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6482.233833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6482.296291] Lustre: 250675:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6482.352519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6482.355026] Lustre: Skipped 1 previous similar message [ 6482.406110] Lustre: lustre-MDT0001: new disk, initializing [ 6482.497774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6482.529685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6482.533671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6486.060894] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6492.507689] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6496.506710] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6502.678822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6502.740311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6502.925783] Lustre: lustre-OST0000: new disk, initializing [ 6502.929433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6502.993758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.053855] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6508.581474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6508.592297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6508.687764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6512.788682] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6514.556596] LustreError: 250681:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6522.085182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6522.169528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6522.343537] Lustre: lustre-OST0001: new disk, initializing [ 6522.351159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6524.648165] LustreError: 250681:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6524.676698] LustreError: 250681:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6526.544406] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6531.647474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6531.651920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6531.719422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6532.583934] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6533.907673] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6534.743756] LustreError: 250681:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6536.680927] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.684959] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6536.697675] Lustre: Skipped 2 previous similar messages [ 6536.718931] Lustre: Skipped 4 previous similar messages [ 6538.423798] LustreError: 254168:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6538.430555] LustreError: 254168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6538.485484] Lustre: server umount lustre-OST0001 complete [ 6541.801944] LustreError: 252406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.913293] LustreError: 252407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.935106] LustreError: 252407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6554.592109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6554.601380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6554.611804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6554.614398] Lustre: Skipped 1 previous similar message [ 6557.169614] LustreError: 252408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.178751] LustreError: 252408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6562.280241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6562.291226] Lustre: Skipped 4 previous similar messages [ 6567.396783] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6567.444197] LustreError: 254390:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6567.449053] LustreError: 254390:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6567.526639] Lustre: server umount lustre-OST0000 complete [ 6572.517834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6572.524637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6572.525069] Lustre: Skipped 2 previous similar messages [ 6572.532296] Lustre: Skipped 3 previous similar messages [ 6577.001271] Lustre: server umount lustre-MDT0000 complete [ 6577.632749] LustreError: 251879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.652566] LustreError: 251879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6580.676936] LustreError: 253424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338476 with bad export cookie 2579779564246602906 [ 6580.683717] LustreError: 253424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6580.684836] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6586.929545] LustreError: 254795:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6586.933964] LustreError: 254795:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6587.092849] Lustre: server umount lustre-MDT0001 complete [ 6597.620384] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6601.179137] Key type lgssc unregistered [ 6601.601528] LNet: 255327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6601.612638] LNetError: 255327:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6601.639170] LNet: Removed LNI 192.168.202.119@tcp [ 6602.707324] Key type .llcrypt unregistered [ 6602.712147] Key type ._llcrypt unregistered [ 6636.280038] Key type ._llcrypt registered [ 6636.281245] Key type .llcrypt registered [ 6636.396985] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 6649.506204] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6650.529415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6650.812799] alg: No test for adler32 (adler32-zlib) [ 6651.876164] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 6652.148798] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6653.831310] Key type lgssc registered [ 6654.777091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6662.476602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6667.547579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6672.510679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.525578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.998463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6683.051950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6683.088921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.366453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6684.395245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6684.452916] Lustre: lustre-MDT0000: new disk, initializing [ 6684.529912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.558548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6688.250661] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6697.254229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6697.329657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6697.418122] Lustre: 259275:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6697.493899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6697.496798] Lustre: Skipped 1 previous similar message [ 6697.608719] Lustre: lustre-MDT0001: new disk, initializing [ 6697.719215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6697.776282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6697.784435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6700.829561] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6707.530858] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6710.836958] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6716.525085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6716.592244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6716.864354] Lustre: lustre-OST0000: new disk, initializing [ 6716.873912] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6716.977347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.453417] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6726.756490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6726.773431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6726.881439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6730.306543] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6734.119273] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6734.372380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6738.569864] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6738.773353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6742.503403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6742.504212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.520933] LustreError: Skipped 1 previous similar message [ 6742.532739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6742.556995] Lustre: Skipped 1 previous similar message [ 6746.773165] LustreError: 262211:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6746.909562] Lustre: server umount lustre-OST0000 complete [ 6752.736570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.740475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6752.749114] Lustre: Skipped 2 previous similar messages [ 6752.769140] Lustre: Skipped 1 previous similar message [ 6757.074911] LustreError: 262414:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6757.087850] LustreError: 262414:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6757.167052] Lustre: server umount lustre-MDT0000 complete [ 6760.726184] LustreError: 259268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338656 with bad export cookie 13676448130288149718 [ 6760.728410] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.742335] LustreError: 259268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6760.877474] LustreError: 262616:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6760.881029] LustreError: 262616:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6761.102663] Lustre: server umount lustre-MDT0001 complete [ 6767.908926] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 19:17:41 (1763338661) [ 6773.850830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.245789] LustreError: 263288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.264780] LustreError: 263288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6774.335863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.325851] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6779.369671] LustreError: 263288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.480618] LustreError: 263288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6786.427379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6786.776636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6790.500949] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6795.435681] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6799.791778] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6806.613600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6807.050375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.010265] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6818.267480] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6873.571906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6873.585200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.604686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.621637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.623531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.624671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6876.925039] LustreError: 265614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6876.927463] LustreError: 265614:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6877.077627] Lustre: server umount lustre-OST0000 complete [ 6880.675669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6888.582857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6888.814526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.424388] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 6895.985614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6895.999495] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6898.141372] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6901.159830] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6901.176190] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6937.061162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6937.061843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6937.070017] LustreError: Skipped 1 previous similar message [ 6937.070736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6937.075307] Lustre: Skipped 1 previous similar message [ 6937.077371] Lustre: Skipped 1 previous similar message [ 6941.750118] LustreError: 267279:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6941.758081] LustreError: 267279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6941.877920] Lustre: server umount lustre-OST0000 complete [ 6945.762926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6945.771428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.790452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6951.332987] LustreError: 267480:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6951.357109] LustreError: 267480:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6951.515717] Lustre: server umount lustre-MDT0000 complete [ 6952.426440] LustreError: 264001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.445183] LustreError: 264001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6954.969128] LustreError: 263267:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338850 with bad export cookie 13676448130288150600 [ 6954.976802] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6954.992130] LustreError: 263267:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6955.340981] Lustre: server umount lustre-MDT0001 complete [ 6963.923452] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 6965.874359] Key type lgssc unregistered [ 6966.133197] LNet: 268213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6966.137475] LNetError: 268213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6966.150718] LNet: Removed LNI 192.168.202.119@tcp [ 6966.820680] Key type .llcrypt unregistered [ 6966.823176] Key type ._llcrypt unregistered [ 6985.667587] Key type ._llcrypt registered [ 6985.669438] Key type .llcrypt registered [ 6985.754813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 6986.794850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6986.803821] alg: No test for adler32 (adler32-zlib) [ 6987.825498] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 6988.026673] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 6989.679225] Key type lgssc registered [ 6990.878610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6998.932332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6998.944706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.478802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7004.539880] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7009.762660] LustreError: 269606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7009.787390] LustreError: 269606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7012.490803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7012.954719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7016.597455] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7021.370181] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7024.591297] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7030.000108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7030.248534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.200529] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7039.789820] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7044.588098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7069.161597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7069.173638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.188502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.181978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7070.183481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7070.196122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.615319] LustreError: 272624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7071.723231] Lustre: server umount lustre-OST0000 complete [ 7075.297188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.319959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7075.329900] Lustre: Skipped 2 previous similar messages [ 7080.419741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7080.434048] Lustre: Skipped 1 previous similar message [ 7081.465583] LustreError: 272825:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7081.487745] LustreError: 272825:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7081.637428] Lustre: server umount lustre-MDT0000 complete [ 7085.307789] LustreError: 269587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338980 with bad export cookie 1884307930790424815 [ 7085.314039] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7085.323267] LustreError: 269587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7085.469453] LustreError: 273027:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7085.475221] LustreError: 273027:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7085.655351] Lustre: server umount lustre-MDT0001 complete [ 7098.556821] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7101.493403] Key type lgssc unregistered [ 7101.809523] LNet: 273560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7101.819366] LNetError: 273560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7101.835134] LNet: Removed LNI 192.168.202.119@tcp [ 7102.586472] Key type .llcrypt unregistered [ 7102.591995] Key type ._llcrypt unregistered [ 7120.586174] Key type ._llcrypt registered [ 7120.588399] Key type .llcrypt registered [ 7120.763274] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7122.045789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.104285] alg: No test for adler32 (adler32-zlib) [ 7123.157299] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 7123.338204] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7124.999193] Key type lgssc registered [ 7126.350848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7134.391384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7134.410247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.903740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.147186] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7145.872569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7146.228331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.643840] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7154.215145] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7157.713816] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7164.703215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7165.026494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7170.051740] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7174.986783] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7182.457084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7197.154237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7197.158170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.173842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7198.179870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.184786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7198.185997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7200.193139] LustreError: 277184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7200.348420] Lustre: server umount lustre-OST0000 complete [ 7207.910453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7207.919416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.933325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.937131] Lustre: Skipped 1 previous similar message [ 7210.596899] LustreError: 277385:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7210.611034] LustreError: 277385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7210.674073] Lustre: server umount lustre-MDT0000 complete [ 7213.545322] LustreError: 274766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.559473] LustreError: 274766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7213.797194] LustreError: 274749:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339109 with bad export cookie 18426533138121355095 [ 7213.800945] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7213.913222] LustreError: 277586:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7213.915435] LustreError: 277586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7214.052237] Lustre: server umount lustre-MDT0001 complete [ 7221.787824] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7224.260976] Key type lgssc unregistered [ 7224.519651] LNet: 278116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7224.523649] LNetError: 278116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7224.533613] LNet: Removed LNI 192.168.202.119@tcp [ 7225.188212] Key type .llcrypt unregistered [ 7225.192821] Key type ._llcrypt unregistered [ 7244.724415] Key type ._llcrypt registered [ 7244.726182] Key type .llcrypt registered [ 7244.817772] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7245.680025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7245.736961] alg: No test for adler32 (adler32-zlib) [ 7246.800722] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 7247.115083] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7248.816707] Key type lgssc registered [ 7249.790835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7255.839985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7255.868697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7257.222080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.467483] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7266.788938] LustreError: 279492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7266.818789] LustreError: 279492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7268.811098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7269.312218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7273.371857] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7279.730823] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7284.161938] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7290.971485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7291.432631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7296.369460] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7301.695199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7301.880900] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7325.153444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.157593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.172442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.202131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7327.212515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7327.230248] Lustre: Skipped 1 previous similar message [ 7329.141154] LustreError: 282514:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7329.256701] Lustre: server umount lustre-OST0000 complete [ 7335.904059] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7335.909976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.920279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.750939] LustreError: 282715:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7338.774027] LustreError: 282715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7338.927341] Lustre: server umount lustre-MDT0000 complete [ 7342.493652] LustreError: 281113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339237 with bad export cookie 5036122046498544462 [ 7342.494937] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7342.505406] LustreError: 281113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7342.652959] LustreError: 282916:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7342.658843] LustreError: 282916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7342.818139] Lustre: server umount lustre-MDT0001 complete [ 7353.691988] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7356.430342] Key type lgssc unregistered [ 7356.742564] LNet: 283448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7356.749845] LNetError: 283448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7356.762133] LNet: Removed LNI 192.168.202.119@tcp [ 7357.622408] Key type .llcrypt unregistered [ 7357.630703] Key type ._llcrypt unregistered [ 7373.814953] Key type ._llcrypt registered [ 7373.817216] Key type .llcrypt registered [ 7373.907800] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7374.718187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.906301] alg: No test for adler32 (adler32-zlib) [ 7376.090550] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 7376.293971] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7377.935243] Key type lgssc registered [ 7378.796374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7385.800989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7385.810717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.215295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.447711] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7397.345301] LustreError: 284679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7397.377570] LustreError: 284679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7399.600316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7400.089255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7403.297391] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7409.004831] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.738274] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7420.307117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7420.706327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7426.835289] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7432.058556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7433.122576] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7445.480143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7445.483384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7445.513862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7446.846976] LustreError: 287078:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7447.019774] Lustre: server umount lustre-OST0000 complete [ 7451.110736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7451.123718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7451.159922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7452.640380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7452.640750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.760905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7457.763733] Lustre: Skipped 2 previous similar messages [ 7459.817432] LustreError: 287279:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7459.829726] LustreError: 287279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7459.930420] Lustre: server umount lustre-MDT0000 complete [ 7461.864759] LustreError: 284678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7461.878201] LustreError: 284678:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7464.156334] LustreError: 284632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339359 with bad export cookie 10043993886371790469 [ 7464.166795] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7464.167617] LustreError: 284632:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7464.301414] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7464.310267] LustreError: 287482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7464.545426] Lustre: server umount lustre-MDT0001 complete [ 7476.511923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7479.125735] Key type lgssc unregistered [ 7479.480120] LNet: 288017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7479.497157] LNetError: 288017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7479.518459] LNet: Removed LNI 192.168.202.119@tcp [ 7480.309346] Key type .llcrypt unregistered [ 7480.311142] Key type ._llcrypt unregistered [ 7491.880649] Key type ._llcrypt registered [ 7491.882102] Key type .llcrypt registered [ 7492.009822] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7509.706860] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7524.528725] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7525.531445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7525.652270] alg: No test for adler32 (adler32-zlib) [ 7526.769570] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 7527.073336] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7528.775196] Key type lgssc registered [ 7529.882581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7538.956962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7545.271863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7550.766106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7557.580775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7564.435495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7564.496528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7564.517265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.801772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7565.828932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7565.883641] Lustre: lustre-MDT0000: new disk, initializing [ 7565.967625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7566.003652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7569.590606] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7579.901324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7579.984752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7580.103831] Lustre: 291531:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7580.179933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7580.199443] Lustre: Skipped 1 previous similar message [ 7580.343205] Lustre: lustre-MDT0001: new disk, initializing [ 7580.507188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7580.572936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7580.593459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7585.332909] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7593.408283] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7598.059248] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7605.238709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7605.312610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7605.608948] Lustre: lustre-OST0000: new disk, initializing [ 7605.619709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7605.756644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.656860] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7614.085093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7614.101518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7614.217203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7617.281651] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7620.725693] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7620.883859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7624.635566] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7624.800285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7629.792897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7629.795927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.808851] LustreError: Skipped 1 previous similar message [ 7629.820561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7632.509129] LustreError: 294469:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7632.664366] Lustre: server umount lustre-OST0000 complete [ 7636.960853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7636.963320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.978952] Lustre: Skipped 1 previous similar message [ 7636.983432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.992864] Lustre: Skipped 1 previous similar message [ 7640.033693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7640.034037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.685743] LustreError: 294671:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7642.693747] LustreError: 294671:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7642.767618] Lustre: server umount lustre-MDT0000 complete [ 7646.485992] LustreError: 291520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339541 with bad export cookie 4323801129863882597 [ 7646.488278] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7646.561519] LustreError: 291520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7646.770181] LustreError: 294873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7646.773605] LustreError: 294873:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7647.010109] Lustre: server umount lustre-MDT0001 complete [ 7654.935557] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:32:28 (1763339548) [ 7661.939602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.343574] LustreError: 295542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.369196] LustreError: 295542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7662.440209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.366815] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7667.691318] LustreError: 295542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7671.783455] LustreError: 295543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.421996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7675.772558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7679.621278] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7684.854497] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7688.588328] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7694.621549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7694.991657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.571621] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7705.687927] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7721.956714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.966086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.996259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.998849] Lustre: Skipped 1 previous similar message [ 7724.922156] LustreError: 297816:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7724.933844] LustreError: 297816:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7725.102596] Lustre: server umount lustre-OST0000 complete [ 7732.196153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7732.213382] LustreError: Skipped 1 previous similar message [ 7732.223126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.233843] Lustre: Skipped 1 previous similar message [ 7732.277684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.280446] Lustre: Skipped 2 previous similar messages [ 7735.005424] LustreError: 298017:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7735.015454] LustreError: 298017:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7735.096126] Lustre: server umount lustre-MDT0000 complete [ 7738.290656] LustreError: 295524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339633 with bad export cookie 4323801129863883479 [ 7738.294989] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7738.308750] LustreError: 295524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7738.721181] Lustre: server umount lustre-MDT0001 complete [ 7748.845884] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 7751.521384] Key type lgssc unregistered [ 7751.855411] LNet: 298753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7751.863181] LNetError: 298753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7751.914703] LNet: Removed LNI 192.168.202.119@tcp [ 7752.715506] Key type .llcrypt unregistered [ 7752.722360] Key type ._llcrypt unregistered [ 7778.822540] Key type ._llcrypt registered [ 7778.823945] Key type .llcrypt registered [ 7778.924962] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7792.327882] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 7793.467934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7793.634367] alg: No test for adler32 (adler32-zlib) [ 7794.741674] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 7795.016227] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 7796.703190] Key type lgssc registered [ 7797.990730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7807.009931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7814.008973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7820.160604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7827.070142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7832.395360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7837.634957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7843.442367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7843.486860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7843.501914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7844.714960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7844.753782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7844.809822] Lustre: lustre-MDT0000: new disk, initializing [ 7844.881357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7844.894923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7848.400983] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7860.520878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7860.596677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7860.692733] Lustre: 302563:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7860.763133] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7860.768764] Lustre: Skipped 1 previous similar message [ 7860.842323] Lustre: lustre-MDT0001: new disk, initializing [ 7860.940416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7860.978609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7861.001234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7865.603182] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7873.968821] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7877.777927] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7885.060796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7885.191579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7885.448367] Lustre: lustre-OST2710: new disk, initializing [ 7885.451537] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7885.565630] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7890.966392] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7893.567677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7893.576742] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7893.654445] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7898.762076] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7905.650562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7905.739112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7905.889759] Lustre: lustre-OST03e8: new disk, initializing [ 7905.898539] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7906.028418] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7912.093141] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 7914.622748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7914.646654] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7914.800073] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7919.552969] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7928.037455] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7928.374839] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7933.231538] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7933.535645] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7938.136380] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7938.496789] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7942.933127] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7943.247567] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7948.256308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7948.263204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.285537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7950.047347] LustreError: 306910:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7950.215503] Lustre: server umount lustre-MDT0000 complete [ 7950.821112] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7950.836032] Lustre: Skipped 1 previous similar message [ 7953.963776] LustreError: 302553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339849 with bad export cookie 5294766797004675280 [ 7953.971769] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7953.985361] LustreError: 302553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7954.143986] LustreError: 307111:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7954.148893] LustreError: 307111:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7954.344235] Lustre: server umount lustre-MDT0001 complete [ 7968.540845] LustreError: 307312:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7968.544801] LustreError: 307312:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7968.616908] Lustre: server umount lustre-OST2710 complete [ 7971.817579] Lustre: 300339:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763339851/real 1763339851] req@00000000bf2fc770 x1848995678848000/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763339867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7971.844972] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7971.852891] Lustre: Skipped 1 previous similar message [ 7972.242929] LustreError: 307514:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7972.245917] LustreError: 307514:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7972.401679] Lustre: server umount lustre-OST03e8 complete [ 7989.997625] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 7999.948378] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8010.446717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8017.658983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8025.415645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8033.847852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8041.633374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8041.720666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8042.075798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8042.115491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8042.211515] Lustre: lustre-MDT0000: new disk, initializing [ 8042.334257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8042.357347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8047.548652] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8059.128685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8059.229546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8059.303777] Lustre: 310312:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8059.352148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8059.354931] Lustre: Skipped 1 previous similar message [ 8059.434487] Lustre: lustre-MDT0001: new disk, initializing [ 8059.529228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8059.573257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8059.578732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8064.111248] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8073.347324] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8078.177236] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8086.611518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8086.693423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8086.997113] Lustre: lustre-OST0000: new disk, initializing [ 8087.007351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8087.154265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.287250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8093.305837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8093.470665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8093.927489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8101.696647] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8105.719053] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8105.952730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8109.793459] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8109.973751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8114.144120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.146982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.149696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.187661] Lustre: Skipped 1 previous similar message [ 8114.197787] Lustre: Skipped 1 previous similar message [ 8117.764440] LustreError: 313251:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8117.776403] LustreError: 313251:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8117.934291] Lustre: server umount lustre-OST0000 complete [ 8124.385282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.409624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8124.423344] Lustre: Skipped 1 previous similar message [ 8127.636662] LustreError: 313455:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8127.653123] LustreError: 313455:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8127.791740] Lustre: server umount lustre-MDT0000 complete [ 8129.514172] LustreError: 310320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.534125] LustreError: 310320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8131.275740] LustreError: 310304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340026 with bad export cookie 5294766797004676960 [ 8131.281348] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.286126] LustreError: 310304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8131.620372] Lustre: server umount lustre-MDT0001 complete [ 8140.110525] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:40:33 (1763340033) [ 8141.455676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8143.257708] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:40:36 (1763340036) [ 8146.480905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8153.962288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8158.263637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8162.187194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8165.922900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8167.049810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8172.317188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.498652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8172.518518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8172.619248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8175.216506] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8181.910272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8182.026972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8182.072166] Lustre: 315787:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8185.705633] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8190.238268] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8193.545278] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8199.679297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8199.744478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8199.965770] Lustre: Denying initial registration attempt from nid 192.168.202.119@tcp, specified as failover [ 8199.978844] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8199.989885] LustreError: 317407:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8200.001133] LustreError: 317407:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8200.007191] LustreError: 317407:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8200.061458] Lustre: server umount lustre-OST0000 complete [ 8200.063602] LustreError: 317407:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8207.840276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8207.846501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8207.854640] LustreError: Skipped 1 previous similar message [ 8207.868270] Lustre: Skipped 1 previous similar message [ 8207.875201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8207.883600] Lustre: Skipped 1 previous similar message [ 8211.809076] LustreError: 317775:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8211.810862] LustreError: 317775:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8211.931491] Lustre: server umount lustre-MDT0000 complete [ 8212.960331] LustreError: 315800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8212.977033] LustreError: 315800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8215.380448] LustreError: 317776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340110 with bad export cookie 5294766797004677842 [ 8215.385067] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8215.388294] LustreError: 317776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8215.659157] Lustre: server umount lustre-MDT0001 complete [ 8226.344794] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8229.178979] Key type lgssc unregistered [ 8229.456387] LNet: 318509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8229.470974] LNetError: 318509:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8229.485386] LNet: Removed LNI 192.168.202.119@tcp [ 8230.403530] Key type .llcrypt unregistered [ 8230.407164] Key type ._llcrypt unregistered [ 8242.166284] Key type ._llcrypt registered [ 8242.182634] Key type .llcrypt registered [ 8242.312475] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8243.495347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8243.524700] alg: No test for adler32 (adler32-zlib) [ 8244.697464] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 8244.894427] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8246.569233] Key type lgssc registered [ 8247.673930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8254.738068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8258.833221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8262.667850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8266.814893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8267.903312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8283.243105] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8291.330048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8291.348223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.623314] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8292.651780] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8292.769389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.806597] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8305.068356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8305.190194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8305.260899] Lustre: 321273:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8305.717117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8309.661317] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8314.699576] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8318.313918] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8325.894482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8325.982416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8326.148206] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8326.365719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8331.607799] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8337.351755] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8343.525146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.532238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.558208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.036813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8344.038112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.058136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8344.062233] Lustre: Skipped 1 previous similar message [ 8345.722150] LustreError: 323559:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8345.842941] Lustre: server umount lustre-OST0000 complete [ 8351.715822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8351.728668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.735877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.277566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.277981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.762462] LustreError: 323760:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8355.773108] LustreError: 323760:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8355.930648] Lustre: server umount lustre-MDT0000 complete [ 8359.062690] LustreError: 321264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340254 with bad export cookie 2309311657392116567 [ 8359.069458] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8359.075932] LustreError: 321264:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8359.236297] LustreError: 323961:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8359.255313] LustreError: 323961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8359.442718] Lustre: server umount lustre-MDT0001 complete [ 8369.485879] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8372.458610] Key type lgssc unregistered [ 8372.781841] LNet: 324493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8372.785834] LNetError: 324493:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8372.800067] LNet: Removed LNI 192.168.202.119@tcp [ 8373.738938] Key type .llcrypt unregistered [ 8373.740489] Key type ._llcrypt unregistered [ 8398.744533] Key type ._llcrypt registered [ 8398.745584] Key type .llcrypt registered [ 8398.835931] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8399.902321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8399.952766] alg: No test for adler32 (adler32-zlib) [ 8401.036204] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 8401.301726] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8402.943892] Key type lgssc registered [ 8404.017714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8412.070492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8412.097755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.649800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.290711] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8423.907485] LustreError: 325889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8423.925382] LustreError: 325889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8425.915223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8426.339688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.845886] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8435.347192] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8439.045800] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8445.434406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8445.739605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.316243] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8456.388638] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8472.531375] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [ 8472.548151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8472.554963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.597958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8474.599401] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.603014] Lustre: Skipped 2 previous similar messages [ 8474.613235] Lustre: Skipped 1 previous similar message [ 8476.135101] LustreError: 328163:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8476.283500] Lustre: server umount lustre-MDT0000 complete [ 8477.637702] LustreError: 325884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.703755] LustreError: 325871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340375 with bad export cookie 16048979086881112959 [ 8479.714726] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8479.721438] LustreError: 325871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8479.735187] LustreError: 325884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.746813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8479.774723] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8479.777369] Lustre: Skipped 1 previous similar message [ 8482.760099] LustreError: 325884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8482.784978] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [ 8485.977969] LustreError: 328364:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8485.987756] LustreError: 328364:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8486.257735] Lustre: server umount lustre-MDT0001 complete [ 8492.149547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8499.877325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.589072] LustreError: 327501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.596381] LustreError: 327501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8505.664710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.732153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8509.656259] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8514.856405] LustreError: 327500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8514.859634] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8514.889873] LustreError: 327500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8517.397982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8517.717839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8521.964648] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8522.749371] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8528.826724] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8530.899267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8535.265932] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8536.909749] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8541.665993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.669644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.678693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8541.683115] Lustre: Skipped 1 previous similar message [ 8543.208272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8547.667137] LustreError: 330669:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8547.673103] LustreError: 330669:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8547.786477] Lustre: server umount lustre-OST0000 complete [ 8553.440853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.443883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8553.480566] Lustre: Skipped 2 previous similar messages [ 8553.480792] Lustre: Skipped 2 previous similar messages [ 8557.722399] LustreError: 330871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8557.744948] LustreError: 330871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8557.884934] Lustre: server umount lustre-MDT0000 complete [ 8558.560307] LustreError: 329176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8558.569322] LustreError: 329176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8561.483778] LustreError: 328365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340456 with bad export cookie 16048979086881120533 [ 8561.488104] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8561.527970] LustreError: 328365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8561.748511] LustreError: 331072:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 8561.752804] LustreError: 331072:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 8561.915671] Lustre: server umount lustre-MDT0001 complete [ 8571.956168] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8575.472931] Key type lgssc unregistered [ 8575.873952] LNet: 331603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8575.892883] LNetError: 331603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8575.916860] LNet: Removed LNI 192.168.202.119@tcp [ 8577.087677] Key type .llcrypt unregistered [ 8577.089049] Key type ._llcrypt unregistered [ 8603.342915] Key type ._llcrypt registered [ 8603.344344] Key type .llcrypt registered [ 8603.442643] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8604.336681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8604.403228] alg: No test for adler32 (adler32-zlib) [ 8605.578296] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 8605.879237] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8607.543177] Key type lgssc registered [ 8608.636335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8616.206341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8616.220353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.766104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8621.100524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8628.201158] LustreError: 333001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.218677] LustreError: 333001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8629.315695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8629.899114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8634.547615] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8640.162250] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8643.594311] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8648.679261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.683586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8648.700575] Lustre: Skipped 1 previous similar message [ 8653.258530] LustreError: 334392:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8653.441854] Lustre: server umount lustre-MDT0000 complete [ 8653.794232] LustreError: 333001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8653.811460] LustreError: 333001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8654.818718] LustreError: 332995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8654.833950] LustreError: 332995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8656.624564] LustreError: 332981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340552 with bad export cookie 6711910392889336169 [ 8656.634654] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8656.747660] LustreError: 334594:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8656.750745] LustreError: 334594:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8656.913446] Lustre: server umount lustre-MDT0001 complete [ 8670.014032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.333828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8670.415084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8670.711650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.439624] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8685.025395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8685.357897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8685.426668] Lustre: 335364:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8685.716741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8689.921718] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8696.054774] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8699.392250] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8706.009527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8706.191399] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8706.198320] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8706.207420] LustreError: 337025:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 8706.212987] LustreError: 337025:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8706.218625] LustreError: 337025:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8706.275685] Lustre: server umount lustre-OST0000 complete [ 8706.277958] LustreError: 337025:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8714.215070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8714.405229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8714.577894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.677753] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8723.109140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8725.073307] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8733.103358] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8733.165486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8733.218422] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8733.295594] Lustre: lustre-OST0001: new disk, initializing [ 8733.302097] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8733.396279] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8738.399427] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8741.461000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8741.469114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8741.548897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8743.755679] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8746.977060] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8746.980430] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.985205] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8746.985214] Lustre: Skipped 1 previous similar message [ 8746.991289] LustreError: Skipped 2 previous similar messages [ 8747.025757] Lustre: Skipped 1 previous similar message [ 8748.606543] LustreError: 339146:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8748.609839] LustreError: 339146:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8748.691155] Lustre: server umount lustre-OST0001 complete [ 8750.050984] LustreError: 337494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.070213] LustreError: 337494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8753.641056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8753.645334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8753.665742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8753.668229] Lustre: Skipped 1 previous similar message [ 8755.182966] LustreError: 337493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8755.201309] LustreError: 337493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8757.832670] LustreError: 339348:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8757.838786] LustreError: 339348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8757.935915] Lustre: server umount lustre-OST0000 complete [ 8762.851298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8762.866536] LustreError: Skipped 1 previous similar message [ 8762.869190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8762.895155] Lustre: Skipped 1 previous similar message [ 8762.906953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8762.909764] Lustre: Skipped 3 previous similar messages [ 8767.880889] LustreError: 339550:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8767.893242] LustreError: 339550:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8767.991420] Lustre: server umount lustre-MDT0000 complete [ 8770.532132] LustreError: 335370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.545945] LustreError: 335370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8771.115533] LustreError: 335357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340666 with bad export cookie 6711910392889336722 [ 8771.118909] LustreError: 338930:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0001-osc-MDT0001: can't precreate: rc = -5 [ 8771.119535] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8771.132359] LustreError: 335357:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8771.137916] LustreError: 338930:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0001-osc-MDT0001: cannot precreate objects: rc = -5 [ 8771.355511] Lustre: server umount lustre-MDT0001 complete [ 8780.107836] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 8783.094204] Key type lgssc unregistered [ 8783.394040] LNet: 340284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8783.400550] LNetError: 340284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8783.418396] LNet: Removed LNI 192.168.202.119@tcp [ 8784.276558] Key type .llcrypt unregistered [ 8784.281994] Key type ._llcrypt unregistered [ 8814.325836] Key type ._llcrypt registered [ 8814.327830] Key type .llcrypt registered [ 8814.456539] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 8827.055798] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 8827.793731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8827.875610] alg: No test for adler32 (adler32-zlib) [ 8828.889810] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 8829.111972] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 8830.783211] Key type lgssc registered [ 8831.767220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8839.349506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8844.854176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.597132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8857.156225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8864.527670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8864.573651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8864.588811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.860399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8865.900950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8866.005422] Lustre: lustre-MDT0000: new disk, initializing [ 8866.114274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.156615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8870.001756] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8880.356616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8880.422529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8880.500670] Lustre: 344234:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8880.567412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8880.570116] Lustre: Skipped 1 previous similar message [ 8880.629485] Lustre: lustre-MDT0001: new disk, initializing [ 8880.744714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8880.788237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8880.794853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8884.478531] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8891.666530] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8895.882314] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8903.549882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8903.604539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8903.847228] Lustre: lustre-OST0000: new disk, initializing [ 8903.865740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8903.969730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8909.266848] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 8911.912684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8911.918262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8912.051254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8915.937433] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8919.893844] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8920.090597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8924.069676] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8924.345689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8927.202575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8927.203099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8927.222854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.226605] Lustre: Skipped 1 previous similar message [ 8932.151082] LustreError: 347175:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8932.320303] Lustre: server umount lustre-OST0000 complete [ 8936.930263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8936.942516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.952102] Lustre: Skipped 1 previous similar message [ 8936.954829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.568098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.573133] Lustre: Skipped 3 previous similar messages [ 8946.659465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8946.662608] Lustre: Skipped 1 previous similar message [ 8950.240192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8950.343438] LustreError: 347377:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8950.347976] LustreError: 347377:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8950.496556] Lustre: server umount lustre-MDT0000 complete [ 8954.312323] LustreError: 345015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340849 with bad export cookie 8930546568288573823 [ 8954.316617] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8954.489209] LustreError: 347579:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8954.491499] LustreError: 347579:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8954.636833] Lustre: server umount lustre-MDT0001 complete [ 8963.031278] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 19:54:16 (1763340856) [ 8968.621081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8974.748448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8996.700735] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9006.641941] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9014.935737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9021.081440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9027.877827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9035.122703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9042.576212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9042.655880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.852858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9042.884292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9042.943747] Lustre: lustre-MDT0000: new disk, initializing [ 9043.037644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.056814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9046.576730] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9056.448828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9056.511829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9056.579194] Lustre: 351356:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9056.649418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9056.653053] Lustre: Skipped 1 previous similar message [ 9056.734938] Lustre: lustre-MDT0001: new disk, initializing [ 9056.835930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9056.879943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9056.900911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9061.065314] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9069.105475] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9072.385959] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9079.572957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9079.685531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9080.216266] Lustre: lustre-OST0000: new disk, initializing [ 9080.221513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9080.360288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9086.355251] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9090.622226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9090.634642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9090.758149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9092.728794] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9096.544552] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9096.744597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9100.620833] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9100.865977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9106.403463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.404457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9106.407496] Lustre: Skipped 1 previous similar message [ 9106.408118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9116.642137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9116.644408] Lustre: Skipped 3 previous similar messages [ 9117.151695] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9117.223100] LustreError: 354298:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9117.236089] LustreError: 354298:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9117.386743] Lustre: server umount lustre-OST0000 complete [ 9121.768895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9121.774212] Lustre: Skipped 1 previous similar message [ 9135.075536] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9135.210623] LustreError: 354501:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9135.220319] LustreError: 354501:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9135.360025] Lustre: server umount lustre-MDT0000 complete [ 9137.120476] LustreError: 351364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9137.137776] LustreError: 351364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9138.932592] LustreError: 351349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341034 with bad export cookie 8930546568288574698 [ 9138.934108] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9138.945481] LustreError: 351349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9139.346543] Lustre: server umount lustre-MDT0001 complete [ 9147.120113] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:57:20 (1763341040) [ 9153.590490] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:57:27 (1763341047) [ 9161.125759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.657412] LustreError: 355718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9161.738348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9165.728959] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9166.821487] LustreError: 355718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9171.941243] LustreError: 355717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.875909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9178.531203] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9183.312867] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9186.156188] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9192.464702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9198.072355] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9202.793853] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9214.373711] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9216.003523] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9217.471593] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9218.953415] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9220.758593] Lustre: Failing over lustre-MDT0000 [ 9220.930466] LustreError: 358175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9220.934123] LustreError: 358175:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9221.002204] Lustre: server umount lustre-MDT0000 complete [ 9221.088391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9221.096901] LustreError: Skipped 1 previous similar message [ 9221.103912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9221.121403] LustreError: 357874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.257699] LustreError: 355713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.280668] LustreError: 355713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9239.007192] Lustre: 342446:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341118/real 1763341118] req@00000000db087885 x1848996763115392/t0(0) o400->MGC192.168.202.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763341134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9239.034283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.059568] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9249.251964] LustreError: 355713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.280495] LustreError: 355713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9249.488306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.504204] Lustre: Skipped 2 previous similar messages [ 9249.572555] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9249.729940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9253.024913] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9254.899593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9254.925731] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9255.007129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9259.586780] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9260.975345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9264.182352] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9265.780324] Lustre: Failing over lustre-MDT0000 [ 9265.915399] LustreError: 359261:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9265.919900] LustreError: 359261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9266.003958] Lustre: server umount lustre-MDT0000 complete [ 9268.194217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.202494] Lustre: Skipped 4 previous similar messages [ 9282.679328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.773853] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9283.072516] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9285.575896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9285.764424] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9288.176142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9288.182358] Lustre: Skipped 2 previous similar messages [ 9288.188150] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9288.216370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9291.982615] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9293.718566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9303.522313] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.523192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9303.529984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9303.529995] Lustre: Skipped 6 previous similar messages [ 9303.536226] Lustre: Skipped 1 previous similar message [ 9304.669943] LustreError: 360294:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9304.675836] LustreError: 360294:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9304.756141] Lustre: server umount lustre-MDT0000 complete [ 9308.360633] LustreError: 360295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341203 with bad export cookie 8930546568288577407 [ 9308.370236] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9308.374684] LustreError: 360295:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9308.718929] Lustre: server umount lustre-MDT0001 complete [ 9323.033595] Lustre: server umount lustre-OST0000 complete [ 9337.779976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.201849] LustreError: 361528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.215805] LustreError: 361528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 9338.270545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.274140] Lustre: Skipped 1 previous similar message [ 9342.324451] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9350.266580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9353.315969] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9357.534519] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9360.436446] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9366.213373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9372.079845] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9377.796374] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9383.998333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9394.146175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.158657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9394.160117] Lustre: Skipped 2 previous similar messages [ 9394.168672] Lustre: Skipped 3 previous similar messages [ 9398.138114] LustreError: 363806:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9398.151038] LustreError: 363806:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9398.303741] Lustre: server umount lustre-OST0000 complete [ 9401.826160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9416.160396] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9416.386287] Lustre: server umount lustre-MDT0000 complete [ 9418.735215] LustreError: 361528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9418.743832] LustreError: 361528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9420.232810] LustreError: 361509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341315 with bad export cookie 8930546568288578135 [ 9420.234929] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9420.252866] LustreError: 361509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9420.734754] Lustre: server umount lustre-MDT0001 complete [ 9430.647833] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9433.116172] Key type lgssc unregistered [ 9433.352845] LNet: 364743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9433.362792] LNetError: 364743:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9433.385507] LNet: Removed LNI 192.168.202.119@tcp [ 9434.092354] Key type .llcrypt unregistered [ 9434.094276] Key type ._llcrypt unregistered [ 9456.065808] Key type ._llcrypt registered [ 9456.067393] Key type .llcrypt registered [ 9456.159857] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9457.001186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9457.033857] alg: No test for adler32 (adler32-zlib) [ 9458.187199] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 9458.432193] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9460.127211] Key type lgssc registered [ 9461.239869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9468.489033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9468.511500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.885346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9473.193761] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9480.166499] LustreError: 366233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9480.182883] LustreError: 366233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9480.699410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9481.052338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9484.872174] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9489.991200] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9493.117572] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9499.141599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9499.523769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.082095] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9510.907589] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9514.869989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9525.221780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9525.224035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9525.235260] Lustre: Skipped 2 previous similar messages [ 9525.242151] Lustre: Skipped 1 previous similar message [ 9526.147698] LustreError: 368501:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9526.220377] Lustre: server umount lustre-MDT0000 complete [ 9529.899257] LustreError: 366213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341425 with bad export cookie 9505201479597794143 [ 9529.909994] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.041527] LustreError: 368703:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9530.050802] LustreError: 368703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9530.229168] Lustre: server umount lustre-MDT0001 complete [ 9544.220351] LustreError: 368904:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9544.222695] LustreError: 368904:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9544.484173] Lustre: server umount lustre-OST0000 complete [ 9558.078933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.578071] LustreError: 369636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9558.652674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9562.421399] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9564.130067] LustreError: 369636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9569.252492] LustreError: 369636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9571.230990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9571.700205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9575.595412] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9580.918869] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9584.205328] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9591.287302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9591.525797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.813927] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9600.808091] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9603.823274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9617.892058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9617.907745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.924150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.928845] Lustre: Skipped 1 previous similar message [ 9619.424788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9619.427794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.511063] LustreError: 371983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9622.514268] LustreError: 371983:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9622.597240] Lustre: server umount lustre-MDT0000 complete [ 9624.546128] LustreError: 369630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9624.558297] LustreError: 369630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9625.802794] LustreError: 369616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341521 with bad export cookie 9505201479597795256 [ 9625.812089] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.817712] LustreError: 369616:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9626.154533] Lustre: server umount lustre-MDT0001 complete [ 9640.022959] LustreError: 372386:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9640.033394] LustreError: 372386:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9640.190431] Lustre: server umount lustre-OST0000 complete [ 9653.390131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9653.816482] LustreError: 373216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.894935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.210821] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9663.893823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9666.989705] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9670.739311] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9673.358658] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9678.826583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9680.201242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9684.052878] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9688.586822] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9694.688175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.694632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9694.707049] Lustre: Skipped 2 previous similar messages [ 9694.707622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9694.722472] Lustre: Skipped 3 previous similar messages [ 9698.692277] LustreError: 375458:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9698.696135] LustreError: 375458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9698.775606] Lustre: server umount lustre-OST0000 complete [ 9704.928148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9704.935246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.951699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.956059] Lustre: Skipped 1 previous similar message [ 9704.976446] Lustre: Skipped 1 previous similar message [ 9708.914346] Lustre: server umount lustre-MDT0000 complete [ 9710.063247] LustreError: 373216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9710.088501] LustreError: 373216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9712.846323] LustreError: 374833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341608 with bad export cookie 9505201479597796271 [ 9712.852742] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9712.864342] LustreError: 374833:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9713.188260] Lustre: server umount lustre-MDT0001 complete [ 9722.331033] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9725.073661] Key type lgssc unregistered [ 9725.366581] LNet: 376396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9725.382635] LNetError: 376396:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9725.397724] LNet: Removed LNI 192.168.202.119@tcp [ 9726.412715] Key type .llcrypt unregistered [ 9726.415066] Key type ._llcrypt unregistered [ 9748.189833] Key type ._llcrypt registered [ 9748.191125] Key type .llcrypt registered [ 9748.267595] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9749.159779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9749.215473] alg: No test for adler32 (adler32-zlib) [ 9750.242828] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 9750.441376] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9752.127181] Key type lgssc registered [ 9753.255793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9760.995770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9761.009666] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9761.014498] LustreError: 377810:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9761.026756] LustreError: 377810:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9761.031795] LustreError: 377810:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9761.041054] LustreError: 377810:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9761.044878] LustreError: 377810:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9768.715756] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9768.734961] LustreError: 378208:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9768.758954] LustreError: 378208:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9768.765569] LustreError: 378208:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9768.782112] LustreError: 378208:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9768.790270] LustreError: 378208:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9785.713286] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [ 9788.277438] Key type lgssc unregistered [ 9788.680559] LNet: 379194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9788.697311] LNetError: 379194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9788.721806] LNet: Removed LNI 192.168.202.119@tcp [ 9789.815599] Key type .llcrypt unregistered [ 9789.817362] Key type ._llcrypt unregistered [ 9809.245824] Key type ._llcrypt registered [ 9809.247490] Key type .llcrypt registered [ 9809.415566] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [ 9824.294636] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [ 9825.430074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9825.463581] alg: No test for adler32 (adler32-zlib) [ 9826.460322] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 9826.604088] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [ 9828.223186] Key type lgssc registered [ 9829.175741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9837.883909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9844.519909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9850.919649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9856.431713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9861.960154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9862.001349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9862.020460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.266237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9863.299106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9863.391796] Lustre: lustre-MDT0000: new disk, initializing [ 9863.506041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.524686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9866.803225] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9875.014191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9875.102053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9875.188615] Lustre: 382324:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9875.242872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9875.249158] Lustre: Skipped 1 previous similar message [ 9875.335569] Lustre: lustre-MDT0001: new disk, initializing [ 9875.439588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.493495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9875.497585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9878.786833] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9884.703349] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9887.780779] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9892.884649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9892.949942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9893.168593] Lustre: lustre-OST0000: new disk, initializing [ 9893.176758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9893.269060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9898.733513] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9902.119138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9902.126739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9902.254835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9906.504826] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9911.181106] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9911.406875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9915.376955] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9915.543813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9917.411901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9917.415510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.439229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9917.924845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9917.934166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.944257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9923.060147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9923.069727] Lustre: Skipped 1 previous similar message [ 9923.158802] LustreError: 385267:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9923.311640] Lustre: server umount lustre-OST0000 complete [ 9926.629392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9926.637791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9926.654301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9932.771789] LustreError: 385470:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9932.775096] LustreError: 385470:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9932.863606] Lustre: server umount lustre-MDT0000 complete [ 9936.190839] LustreError: 382316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341831 with bad export cookie 6556553392147473841 [ 9936.198540] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9936.363434] LustreError: 385671:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9936.365937] LustreError: 385671:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9936.530930] Lustre: server umount lustre-MDT0001 complete [ 9944.293132] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9945.846683] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:10:39 (1763341839) [ 9952.306878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9952.663910] LustreError: 386393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.689779] LustreError: 386393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9952.735691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9956.124351] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9957.864961] LustreError: 386393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9962.977621] LustreError: 386393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.926472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9964.301556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9967.597597] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9972.292773] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9975.468876] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9981.681260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9982.011193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.636352] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [ 9991.696691] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9998.126920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9998.177748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9998.343639] Lustre: lustre-OST0001: new disk, initializing [ 9998.356349] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9998.463848] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10003.690720] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10006.106187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10006.116062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10006.177341] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10010.496508] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10016.742811] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10016.743590] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.749221] LustreError: Skipped 1 previous similar message [10016.750612] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10016.759101] Lustre: Skipped 2 previous similar messages [10016.760452] Lustre: Skipped 2 previous similar messages [10020.163115] LustreError: 389592:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [10020.166219] LustreError: 389592:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10020.276398] Lustre: server umount lustre-OST0001 complete [10021.869074] LustreError: 389342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10021.875245] LustreError: 389342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10026.978336] LustreError: 389342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10026.995670] LustreError: 389342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10037.217607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.221919] LustreError: 387989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10037.223710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10037.223718] Lustre: Skipped 2 previous similar messages [10037.229454] Lustre: Skipped 1 previous similar message [10037.238847] LustreError: 387989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10042.634227] LustreError: 389816:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10042.667989] LustreError: 389816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10042.868585] Lustre: server umount lustre-OST0000 complete [10046.432897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10046.438435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10052.679681] LustreError: 390018:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [10052.687135] LustreError: 390018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10052.772278] Lustre: server umount lustre-MDT0000 complete [10056.376449] LustreError: 386374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341951 with bad export cookie 6556553392147474723 [10056.381136] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10056.390902] LustreError: 386374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10062.806708] Lustre: server umount lustre-MDT0001 complete [10074.500371] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10077.967964] Key type lgssc unregistered [10078.346611] LNet: 390751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10078.362462] LNetError: 390751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10078.390621] LNet: Removed LNI 192.168.202.119@tcp [10079.403206] Key type .llcrypt unregistered [10079.404850] Key type ._llcrypt unregistered [10113.105859] Key type ._llcrypt registered [10113.109346] Key type .llcrypt registered [10113.192939] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10125.069500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10126.073542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10126.173899] alg: No test for adler32 (adler32-zlib) [10127.266975] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [10127.457496] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10129.119185] Key type lgssc registered [10130.128835] Lustre: Echo OBD driver; http://www.lustre.org/ [10138.615973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10145.208930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10151.740224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10158.540328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10165.616471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10165.693400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10165.720811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10167.052488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10167.102911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10167.194888] Lustre: lustre-MDT0000: new disk, initializing [10167.277855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10167.303743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10170.690250] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10181.091686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10181.178262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10181.304243] Lustre: 394681:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10181.383523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10181.389416] Lustre: Skipped 1 previous similar message [10181.514489] Lustre: lustre-MDT0001: new disk, initializing [10181.673563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10181.719386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10181.753833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10185.280209] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10192.473728] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10196.245497] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10202.602979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10202.691459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10203.009501] Lustre: lustre-OST0000: new disk, initializing [10203.013043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10203.138505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10208.533803] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10212.943058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10212.946598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10213.088411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10215.084914] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10219.600719] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10219.835598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10224.459495] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10224.731136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10228.709157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10228.712710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10228.713439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10228.721377] LustreError: Skipped 1 previous similar message [10228.741083] Lustre: Skipped 1 previous similar message [10232.920512] LustreError: 397622:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10233.061509] Lustre: server umount lustre-OST0000 complete [10237.922617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10237.925686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10237.947748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10237.955035] Lustre: Skipped 1 previous similar message [10238.945428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10238.968360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10238.984472] Lustre: Skipped 1 previous similar message [10242.850145] LustreError: 397824:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10242.876226] LustreError: 397824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10242.919633] LustreError: 397824:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000042d2d1bc) refcount nonzero (1) after lock cleanup; forcing cleanup. [10243.065561] Lustre: server umount lustre-MDT0000 complete [10246.842166] LustreError: 394672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342142 with bad export cookie 10787438655434337059 [10246.849113] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10246.872618] LustreError: 394672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10247.029885] LustreError: 398028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10247.048526] LustreError: 398028:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10247.292894] Lustre: server umount lustre-MDT0001 complete [10255.834646] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 20:15:49 (1763342149) [10265.105155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10273.397641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.857593] Lustre: lustre-MDT0000: new disk, initializing [10273.917828] LustreError: 399296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.942946] LustreError: 399296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10273.989250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10277.983957] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10279.407893] LustreError: 399296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.514180] LustreError: 399296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10286.672871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10286.999540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10290.986953] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10296.834443] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10300.534961] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10304.482231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.485282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10304.502872] Lustre: Skipped 1 previous similar message [10308.887690] LustreError: 400639:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10308.900515] LustreError: 400639:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10308.910124] LustreError: 400639:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000ac36689d) refcount nonzero (1) after lock cleanup; forcing cleanup. [10309.120661] Lustre: server umount lustre-MDT0000 complete [10309.602439] LustreError: 400318:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.635719] LustreError: 400318:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10312.773547] LustreError: 400640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342208 with bad export cookie 10787438655434337941 [10312.779457] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10312.782022] LustreError: 400640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10312.932557] LustreError: 400842:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10312.945746] LustreError: 400842:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10313.073188] Lustre: server umount lustre-MDT0001 complete [10322.543816] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:16:56 (1763342216) [10328.227152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10328.533061] LustreError: 401562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.558910] LustreError: 401562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10328.602239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.961640] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10337.762856] LustreError: 401562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10337.771781] LustreError: 401562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10339.644975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10343.764994] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10348.916742] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10352.260909] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10358.511939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10358.751779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10358.754820] Lustre: Skipped 1 previous similar message [10364.239452] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10368.944536] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10380.781575] Lustre: Permanently deactivating lustre-OST0000 [10380.797805] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10386.457202] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10387.906245] LustreError: 404157:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10387.911317] LustreError: 404157:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10387.917241] LustreError: 404157:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10390.013193] LustreError: 404206:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10390.020405] LustreError: 404206:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10397.730853] LustreError: 404303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10397.732747] LustreError: 404303:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10397.855721] Lustre: server umount lustre-OST0000 complete [10400.281051] LustreError: 404455:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10400.283817] LustreError: 404455:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10406.880741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10406.883075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.902139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10406.906090] Lustre: Skipped 2 previous similar messages [10412.077612] Lustre: server umount lustre-MDT0000 complete [10416.124309] LustreError: 401542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342311 with bad export cookie 10787438655434338494 [10416.129538] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10416.139452] LustreError: 401542:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10426.345316] LustreError: 404754:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10426.372176] LustreError: 404754:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [10426.634803] Lustre: server umount lustre-MDT0001 complete [10436.114747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10440.664574] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10443.810796] LustreError: 405599:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10449.857537] Lustre: 405938:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.119@tcp [10451.534141] Lustre: server umount MGS complete [10461.591084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.982827] LustreError: 406664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10462.006726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10462.013452] Lustre: Skipped 1 previous similar message [10462.062413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10465.642763] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10473.193797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10473.512251] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10477.251781] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10482.530473] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10485.382460] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10487.002961] Lustre: Permanently reactivating lustre-OST0000 [10487.015995] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10487.019175] Lustre: Skipped 2 previous similar messages [10514.407076] LustreError: 408221:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10523.044254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10530.429780] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10536.672299] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10545.126937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10545.131304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10545.132731] LustreError: Skipped 1 previous similar message [10545.133402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10545.133406] Lustre: Skipped 2 previous similar messages [10545.148793] Lustre: Skipped 2 previous similar messages [10549.261262] LustreError: 409793:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10549.265653] LustreError: 409793:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [10549.406452] Lustre: server umount lustre-OST0000 complete [10555.360881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10564.577991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10564.580670] Lustre: Skipped 5 previous similar messages [10567.135209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10567.361796] Lustre: server umount lustre-MDT0000 complete [10569.697862] LustreError: 408608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.714364] LustreError: 408608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10571.396897] LustreError: 406643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342466 with bad export cookie 10787438655434339453 [10571.402727] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10571.412535] LustreError: 406643:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10571.862984] Lustre: server umount lustre-MDT0001 complete [10582.240644] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10585.442388] Key type lgssc unregistered [10585.782614] LNet: 410729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10585.799500] LNetError: 410729:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10585.820848] LNet: Removed LNI 192.168.202.119@tcp [10586.785311] Key type .llcrypt unregistered [10586.787132] Key type ._llcrypt unregistered [10609.547462] Key type ._llcrypt registered [10609.549878] Key type .llcrypt registered [10609.659931] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [10624.223703] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10625.463703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10625.483666] alg: No test for adler32 (adler32-zlib) [10626.548628] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [10626.791905] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10628.447329] Key type lgssc registered [10629.504227] Lustre: Echo OBD driver; http://www.lustre.org/ [10638.266510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10645.742727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10652.418100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10659.657728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10666.243675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10666.296905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10666.311939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10667.629497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10667.673380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10667.783897] Lustre: lustre-MDT0000: new disk, initializing [10667.940358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10667.972428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10672.368354] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10682.988498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10683.051318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10683.126498] Lustre: 413859:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10683.200426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10683.204021] Lustre: Skipped 1 previous similar message [10683.263924] Lustre: lustre-MDT0001: new disk, initializing [10683.389513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10683.425858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10683.440530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10687.145286] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10694.527034] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10697.873625] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10704.621615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10704.711116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10704.984653] Lustre: lustre-OST0000: new disk, initializing [10705.002113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10705.101732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10710.874603] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10711.102447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10711.111545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10711.245544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10718.672519] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10722.997655] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10723.265622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10727.032602] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10727.227234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10732.001669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10732.002983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10732.019135] LustreError: Skipped 1 previous similar message [10732.022956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10735.165622] LustreError: 416801:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10735.353767] Lustre: server umount lustre-OST0000 complete [10739.681481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10739.684593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.720885] Lustre: Skipped 1 previous similar message [10739.723414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10739.734797] Lustre: Skipped 1 previous similar message [10742.240345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.244603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10745.087902] LustreError: 417003:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10745.102768] LustreError: 417003:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10745.238732] Lustre: server umount lustre-MDT0000 complete [10749.168952] LustreError: 413896:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342644 with bad export cookie 16019837100180444349 [10749.175455] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10749.435720] LustreError: 417205:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10749.446418] LustreError: 417205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10749.724492] Lustre: server umount lustre-MDT0001 complete [10759.373105] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10760.931580] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:24:14 (1763342654) [10767.580486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.094616] LustreError: 417923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.125390] LustreError: 417923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10768.196203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10772.269851] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10773.482234] LustreError: 417923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.574500] LustreError: 417923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.768523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10781.098620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10785.250636] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10790.261154] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10794.207563] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10801.769109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10802.023411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10807.549208] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10813.776165] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10816.188532] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10818.916078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10824.670396] LustreError: 420362:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10824.676935] LustreError: 420362:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10824.900696] Lustre: server umount lustre-OST0000 complete [10832.353046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10832.365882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10832.397839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10832.400352] Lustre: Skipped 1 previous similar message [10839.012595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10839.015297] Lustre: Skipped 2 previous similar messages [10839.035266] LustreError: 420563:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10839.048939] LustreError: 420563:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10839.137664] Lustre: server umount lustre-MDT0000 complete [10842.591184] Lustre: 412073:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342722/real 1763342722] req@000000009693892e x1848998648413056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763342738 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10842.610758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10842.617319] Lustre: Skipped 1 previous similar message [10842.624291] LustreError: 417922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.635373] LustreError: 417922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10843.190790] LustreError: 417903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342738 with bad export cookie 16019837100180445231 [10843.197415] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10843.213460] LustreError: 417903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10843.710265] Lustre: server umount lustre-MDT0001 complete [10853.864850] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10856.609614] Key type lgssc unregistered [10856.958496] LNet: 421299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10856.968520] LNetError: 421299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10856.989872] LNet: Removed LNI 192.168.202.119@tcp [10857.756786] Key type .llcrypt unregistered [10857.761492] Key type ._llcrypt unregistered [10880.514919] Key type ._llcrypt registered [10880.516418] Key type .llcrypt registered [10880.600849] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [10880.874634] Key type .llcrypt unregistered [10880.876362] Key type ._llcrypt unregistered [10897.441507] Key type ._llcrypt registered [10897.442971] Key type .llcrypt registered [10897.613126] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [10898.505792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10898.533339] alg: No test for adler32 (adler32-zlib) [10899.583499] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [10899.760773] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [10901.399400] Key type lgssc registered [10902.527395] Lustre: Echo OBD driver; http://www.lustre.org/ [10910.444964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10910.481931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.163996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.929762] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10922.477312] LustreError: 423471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.513381] LustreError: 423471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10925.334636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10925.645775] LustreError: 423471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.661716] LustreError: 423471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10925.676562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10930.667585] LustreError: 423466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.703321] LustreError: 423466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10932.105986] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10935.783607] LustreError: 424434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.820941] LustreError: 424434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10936.838692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10938.758374] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10941.923527] LustreError: 423466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.937449] LustreError: 423466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10945.454977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10945.917994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10949.956692] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [10959.621655] LustreError: 425524:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10959.748413] Lustre: server umount lustre-OST0000 complete [10965.988092] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10965.992722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.001304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10966.524432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.531157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10966.541883] Lustre: Skipped 1 previous similar message [10971.621899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.661546] LustreError: 425725:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10972.674749] LustreError: 425725:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10972.756947] Lustre: server umount lustre-MDT0000 complete [10976.234094] LustreError: 423453:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342871 with bad export cookie 4082714695304710737 [10976.260429] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10986.480923] LustreError: 425927:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10986.492166] LustreError: 425927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10986.711260] Lustre: server umount lustre-MDT0001 complete [10998.158964] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11001.302463] Key type lgssc unregistered [11001.609809] LNet: 426462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11001.625536] LNetError: 426462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11001.647551] LNet: Removed LNI 192.168.202.119@tcp [11002.617549] Key type .llcrypt unregistered [11002.619373] Key type ._llcrypt unregistered [11028.089330] Key type ._llcrypt registered [11028.090766] Key type .llcrypt registered [11028.200959] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11029.352673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11029.366344] alg: No test for adler32 (adler32-zlib) [11030.553508] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [11030.886241] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11032.647180] Key type lgssc registered [11033.936918] Lustre: Echo OBD driver; http://www.lustre.org/ [11042.685094] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11042.725318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11058.399364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11063.519234] LustreError: 427837:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.119@tcp: failed processing log, type 4: rc = -110 [11090.207166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.493770] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11103.698994] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11110.713856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.223834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11111.321636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11115.509492] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11116.519598] LustreError: 427864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11116.543893] LustreError: 427864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11121.643405] LustreError: 427864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.185424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11124.713902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11128.859852] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11134.877884] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11139.505691] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11147.232161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11147.237747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11147.260628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11149.237106] LustreError: 430118:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [11149.450550] Lustre: server umount lustre-OST0000 complete [11154.944696] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11154.964848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.998194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11155.242603] LustreError: 430320:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11155.256053] LustreError: 430320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11156.449116] LustreError: 428767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11156.455103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11157.348982] LustreError: 430320:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [11157.367035] LustreError: 430320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11157.421676] Lustre: server umount lustre-MDT0000 complete [11161.569098] LustreError: 428767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.590634] LustreError: 428767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11161.813873] LustreError: 427845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343057 with bad export cookie 16960172871696684186 [11161.822219] LustreError: 427845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11161.823298] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11172.324466] LustreError: 430522:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11172.338842] LustreError: 430522:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11172.569387] Lustre: server umount lustre-MDT0001 complete [11184.966347] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11188.276340] Key type lgssc unregistered [11188.551668] LNet: 431055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11188.575246] LNetError: 431055:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11189.611838] LNet: Removed LNI 192.168.202.119@tcp [11190.571839] Key type .llcrypt unregistered [11190.574461] Key type ._llcrypt unregistered [11212.287281] Key type ._llcrypt registered [11212.288463] Key type .llcrypt registered [11212.407737] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11213.938624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11213.954317] alg: No test for adler32 (adler32-zlib) [11215.084722] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [11215.388354] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11217.064661] Key type lgssc registered [11218.354579] Lustre: Echo OBD driver; http://www.lustre.org/ [11226.180683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11226.201728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11227.677292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11231.041811] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11237.860245] LustreError: 432449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11237.881729] LustreError: 432449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11238.526818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11238.822559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11242.491218] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11248.704621] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11252.291744] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11259.133934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11259.440386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11265.714642] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11270.962198] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11272.771400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11282.909665] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.19@tcp (stopping) [11282.912422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11282.920866] Lustre: Skipped 1 previous similar message [11282.935015] Lustre: Skipped 2 previous similar messages [11286.247085] LustreError: 434699:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11286.357950] Lustre: server umount lustre-MDT0000 complete [11288.000485] LustreError: 432446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.752236] LustreError: 432429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.19@tcp arrived at 1763343184 with bad export cookie 762997339195343415 [11288.772803] LustreError: 432429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11291.061400] LustreError: 432429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343186 with bad export cookie 762997339195342743 [11291.087585] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11291.089331] LustreError: 432429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11293.152865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11293.153996] LustreError: 434323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.181264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11293.203082] LustreError: 434323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11296.160660] LustreError: 434323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.166036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11296.191085] LustreError: 434323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11297.229182] LustreError: 434901:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11297.239399] LustreError: 434901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11297.397221] Lustre: server umount lustre-OST0000 complete [11301.289828] LustreError: 433162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.333221] LustreError: 433162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11303.847944] LustreError: 435255:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11303.850974] LustreError: 435255:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11304.073289] Lustre: server umount lustre-MDT0001 complete [11313.901617] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11316.581317] Key type lgssc unregistered [11316.851878] LNet: 435787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11316.862810] LNetError: 435787:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11316.886869] LNet: Removed LNI 192.168.202.119@tcp [11317.695821] Key type .llcrypt unregistered [11317.700664] Key type ._llcrypt unregistered [11340.309623] Key type ._llcrypt registered [11340.310941] Key type .llcrypt registered [11340.430812] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11342.153304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11342.205267] alg: No test for adler32 (adler32-zlib) [11343.337274] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [11343.566598] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11345.231190] Key type lgssc registered [11346.351824] Lustre: Echo OBD driver; http://www.lustre.org/ [11354.390990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11354.404652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11355.948622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11359.919335] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11366.371431] LustreError: 437182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.400023] LustreError: 437182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11368.895481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11369.389798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11373.095239] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11378.865508] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11382.422906] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11388.764370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11389.115418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11394.131779] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11399.464982] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11401.410098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11410.400931] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11410.414960] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11410.427493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11410.431180] Lustre: Skipped 1 previous similar message [11411.936445] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11411.943300] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11413.859155] LustreError: 439431:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11413.957066] Lustre: server umount lustre-MDT0001 complete [11415.505188] LustreError: 439394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11417.058598] LustreError: 439394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11417.078216] LustreError: 439394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11422.177588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.178858] LustreError: 437182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.180202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11422.196822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.199661] Lustre: Skipped 1 previous similar message [11425.634078] LustreError: 439633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11425.640982] LustreError: 439633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11425.766567] Lustre: server umount lustre-OST0000 complete [11427.296843] LustreError: 437182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.961835] LustreError: 439836:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11429.973887] LustreError: 439836:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11430.219316] Lustre: server umount lustre-MDT0000 complete [11442.985544] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11445.239456] Key type lgssc unregistered [11445.503506] LNet: 440521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11445.508793] LNetError: 440521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11445.523768] LNet: Removed LNI 192.168.202.119@tcp [11446.176224] Key type .llcrypt unregistered [11446.182165] Key type ._llcrypt unregistered [11469.797903] Key type ._llcrypt registered [11469.799225] Key type .llcrypt registered [11469.921962] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [11484.806383] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11486.098249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11486.129713] alg: No test for adler32 (adler32-zlib) [11487.277830] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [11487.539713] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11489.199188] Key type lgssc registered [11490.258708] Lustre: Echo OBD driver; http://www.lustre.org/ [11498.820236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11504.373877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11509.946017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11515.995176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11523.869687] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11531.437140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11537.790952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11537.847840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11537.877622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11539.130635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11539.158858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11539.229458] Lustre: lustre-MDT0000: new disk, initializing [11539.322399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11539.344416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11542.625446] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11550.170781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11555.939662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11556.017545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11556.212630] Lustre: lustre-OST0000: new disk, initializing [11556.216264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11556.219173] Lustre: Skipped 1 previous similar message [11556.327494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11561.127674] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11566.646312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11566.655921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11566.675245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11567.463141] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11575.148551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11581.150700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11581.213383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11581.284920] Lustre: 444500:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11581.366944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11581.416325] Lustre: lustre-MDT0001: new disk, initializing [11581.555417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11581.593054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11581.609472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11584.653301] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11592.639507] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11592.827326] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11596.045243] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11596.256488] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11602.383259] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.19@tcp (stopping) [11604.960626] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11604.966229] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.973525] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11604.977708] Lustre: Skipped 1 previous similar message [11606.246634] LustreError: 447920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11606.394720] Lustre: server umount lustre-MDT0001 complete [11615.208724] LustreError: 444506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11615.237725] LustreError: 444506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11620.328329] LustreError: 444512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.345540] LustreError: 444512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11622.880558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11622.886683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.893305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11622.895867] Lustre: Skipped 1 previous similar message [11625.441399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11625.443118] LustreError: 444512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.171716] LustreError: 448190:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11628.173593] LustreError: 448190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11628.303309] Lustre: server umount lustre-OST0000 complete [11630.561455] LustreError: 444512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11637.477802] LustreError: 448392:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11637.485235] LustreError: 448392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11637.725349] Lustre: server umount lustre-MDT0000 complete [11650.212791] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11652.491802] Key type lgssc unregistered [11652.765894] LNet: 449077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11652.771069] LNetError: 449077:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11653.799485] LNet: Removed LNI 192.168.202.119@tcp [11654.622370] Key type .llcrypt unregistered [11654.624205] Key type ._llcrypt unregistered [11694.434520] Key type ._llcrypt registered [11694.435911] Key type .llcrypt registered [11694.540176] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [11694.827684] Key type .llcrypt unregistered [11694.833266] Key type ._llcrypt unregistered [11709.431650] Key type ._llcrypt registered [11709.432753] Key type .llcrypt registered [11709.564198] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [11710.309159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11710.318465] alg: No test for adler32 (adler32-zlib) [11711.381756] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [11711.623378] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [11713.295220] Key type lgssc registered [11714.238983] Lustre: Echo OBD driver; http://www.lustre.org/ [11722.182146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11728.550910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11734.427893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11740.813213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11740.872057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11740.908051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.137664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11742.164930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11742.272578] Lustre: lustre-MDT0000: new disk, initializing [11742.374587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11742.393997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11746.073726] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11756.371793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11756.442268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11756.563732] Lustre: 452442:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11756.644452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11756.651962] Lustre: Skipped 1 previous similar message [11756.680322] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11756.745863] Lustre: lustre-MDT0001: new disk, initializing [11756.881684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11756.924947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11756.938473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11760.816668] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11768.576594] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11772.856821] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11779.295852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11779.393315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11779.672961] Lustre: lustre-OST0000: new disk, initializing [11779.690408] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11779.789838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.291972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11785.299546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11785.460857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11785.743857] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11792.945366] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11801.060183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.064927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11801.075144] Lustre: Skipped 2 previous similar messages [11801.081058] Lustre: Skipped 1 previous similar message [11804.760931] LustreError: 454887:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11804.886594] Lustre: server umount lustre-MDT0000 complete [11808.040982] LustreError: 452433:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343703 with bad export cookie 6480737666434625765 [11808.046853] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11808.192659] LustreError: 455088:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11808.211987] LustreError: 455088:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11808.420318] Lustre: server umount lustre-MDT0001 complete [11822.036088] LustreError: 455291:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11822.044580] LustreError: 455291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11822.169664] Lustre: server umount lustre-OST0000 complete [11831.260677] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:42:04 (1763343724) [11832.343186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11838.612631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11839.227169] LustreError: 456121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11839.253129] LustreError: 456121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11839.330547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.611987] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11844.580076] LustreError: 456121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.680724] LustreError: 456121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.877885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11851.250171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11855.049177] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11860.751624] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11865.409583] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11872.257280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11872.320853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11872.452654] Lustre: Found index 0 for lustre-OST0000, updating log [11872.723235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11879.419216] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11884.020055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11884.892891] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11891.168228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11891.171262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11891.188575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11891.191898] Lustre: Skipped 1 previous similar message [11891.904224] LustreError: 458377:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11891.912254] LustreError: 458377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11892.086547] Lustre: server umount lustre-OST0000 complete [11894.732529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11897.317063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11897.326199] Lustre: Skipped 1 previous similar message [11897.335602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11897.337724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11897.343566] Lustre: Skipped 1 previous similar message [11902.432428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11902.435053] Lustre: Skipped 2 previous similar messages [11902.782507] LustreError: 458631:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11902.785564] LustreError: 458631:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11902.845835] Lustre: server umount lustre-MDT0000 complete [11906.658829] LustreError: 456102:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343802 with bad export cookie 6480737666434627165 [11906.677290] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11906.683200] LustreError: 456102:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [11907.281490] Lustre: server umount lustre-MDT0001 complete [11917.437610] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:43:30 (1763343810) [11918.731151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11919.843248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11920.852821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11921.850844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11922.786069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11923.802758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11924.665923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11925.600422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11926.762961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11927.874594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11928.904345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11929.976991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11931.369560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11932.513990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11933.512216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11934.506269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11935.504297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11935.969886] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp [11936.643662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11937.651145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11938.737289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11939.855173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11941.150797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11942.222120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11943.254229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11944.166652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11945.090300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11945.966865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11947.010173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11948.160706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11949.140230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11950.203089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11951.216767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11952.261211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11953.395583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11954.413539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11955.663626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11956.822054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11957.875754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11958.821059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11959.905729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11960.931384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11961.947072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11963.001661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11964.055637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.102147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11966.041824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11967.053925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11968.088253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11969.142642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11970.230222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11971.401147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11979.902250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.479309] LustreError: 462232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.564110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11984.658923] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [11985.897474] LustreError: 462232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11993.564971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11998.423355] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12004.134692] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12006.623217] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp [12007.864277] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12014.342044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12014.438274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12014.626370] Lustre: Found index 0 for lustre-OST0000, updating log [12014.879317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12014.882866] Lustre: Skipped 1 previous similar message [12020.556280] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12026.478972] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12027.186217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12031.461925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12031.467166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12031.473065] Lustre: Skipped 1 previous similar message [12031.488363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12034.694174] LustreError: 464499:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12034.698531] LustreError: 464499:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12034.845773] Lustre: server umount lustre-OST0000 complete [12038.563286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12038.577797] Lustre: Skipped 2 previous similar messages [12044.958326] Lustre: server umount lustre-MDT0000 complete [12045.735477] LustreError: 462944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12045.769214] LustreError: 462944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12049.416488] LustreError: 462213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343944 with bad export cookie 6480737666434628243 [12049.421869] LustreError: 462213:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12049.428913] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12049.854549] Lustre: server umount lustre-MDT0001 complete [12052.909636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12067.046354] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp [12070.636500] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12081.746691] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12091.301422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12098.633471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12104.900504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12111.731092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12118.770781] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:46:52 (1763344012) [12135.234737] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12137.617603] Key type lgssc unregistered [12137.870132] LNet: 468476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12137.873735] LNetError: 468476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12268.966974] LNet: Removed LNI 192.168.202.119@tcp [12269.877761] Key type .llcrypt unregistered [12269.883702] Key type ._llcrypt unregistered [12286.196604] Key type ._llcrypt registered [12286.198014] Key type .llcrypt registered [12286.284948] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12287.413038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12287.487436] alg: No test for adler32 (adler32-zlib) [12288.584396] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [12288.931778] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12290.591154] Key type lgssc registered [12291.638751] Lustre: Echo OBD driver; http://www.lustre.org/ [12295.594683] LNet: Added LNI 192.168.202.119@tcp100 [8/256/0/180] [12296.454767] LNet: Added LNI 192.168.202.119@tcp101 [8/256/0/180] [12297.298729] LNet: Added LNI 192.168.202.119@tcp102 [8/256/0/180] [12298.170326] LNet: Added LNI 192.168.202.119@tcp103 [8/256/0/180] [12299.060241] LNet: Added LNI 192.168.202.119@tcp104 [8/256/0/180] [12299.900353] LNet: Added LNI 192.168.202.119@tcp105 [8/256/0/180] [12300.857634] LNet: Added LNI 192.168.202.119@tcp106 [8/256/0/180] [12301.723476] LNet: Added LNI 192.168.202.119@tcp107 [8/256/0/180] [12302.619594] LNet: Added LNI 192.168.202.119@tcp108 [8/256/0/180] [12303.662668] LNet: Added LNI 192.168.202.119@tcp109 [8/256/0/180] [12304.768657] LNet: Added LNI 192.168.202.119@tcp110 [8/256/0/180] [12305.686603] LNet: Added LNI 192.168.202.119@tcp111 [8/256/0/180] [12306.585240] LNet: Added LNI 192.168.202.119@tcp112 [8/256/0/180] [12307.483138] LNet: Added LNI 192.168.202.119@tcp113 [8/256/0/180] [12308.235871] LNet: Added LNI 192.168.202.119@tcp114 [8/256/0/180] [12308.980572] LNet: Added LNI 192.168.202.119@tcp115 [8/256/0/180] [12309.781064] LNet: Added LNI 192.168.202.119@tcp116 [8/256/0/180] [12310.689757] LNet: Added LNI 192.168.202.119@tcp117 [8/256/0/180] [12311.631667] LNet: Added LNI 192.168.202.119@tcp118 [8/256/0/180] [12312.460084] LNet: Added LNI 192.168.202.119@tcp119 [8/256/0/180] [12313.438924] LNet: Added LNI 192.168.202.119@tcp120 [8/256/0/180] [12314.305202] LNet: Added LNI 192.168.202.119@tcp121 [8/256/0/180] [12315.198509] LNet: Added LNI 192.168.202.119@tcp122 [8/256/0/180] [12316.180975] LNet: Added LNI 192.168.202.119@tcp123 [8/256/0/180] [12317.030411] LNet: Added LNI 192.168.202.119@tcp124 [8/256/0/180] [12318.016389] LNet: Added LNI 192.168.202.119@tcp125 [8/256/0/180] [12319.161388] LNet: Added LNI 192.168.202.119@tcp126 [8/256/0/180] [12320.147240] LNet: Added LNI 192.168.202.119@tcp127 [8/256/0/180] [12321.259899] LNet: Added LNI 192.168.202.119@tcp128 [8/256/0/180] [12322.211422] LNet: Added LNI 192.168.202.119@tcp129 [8/256/0/180] [12323.064294] LNet: Added LNI 192.168.202.119@tcp130 [8/256/0/180] [12323.904332] LNet: Added LNI 192.168.202.119@tcp131 [8/256/0/180] [12324.839420] LNet: Added LNI 192.168.202.119@tcp132 [8/256/0/180] [12325.772597] LNet: Added LNI 192.168.202.119@tcp133 [8/256/0/180] [12326.662622] LNet: Added LNI 192.168.202.119@tcp134 [8/256/0/180] [12327.496822] LNet: Added LNI 192.168.202.119@tcp135 [8/256/0/180] [12334.280898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12334.343954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12334.364456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.522877] Lustre: 471489:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [12335.577790] LNetError: 468946:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12335.586653] LNetError: Protocol error connecting to 192.168.202.119@tcp119 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12335.615186] LNetError: Skipped 9 previous similar messages [12335.727177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12335.754758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12335.835745] Lustre: lustre-MDT0000: new disk, initializing [12335.966958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12335.991633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12336.033700] LNet: 3 local NIs in recovery (showing 3): 192.168.202.119@tcp123, 192.168.202.119@tcp129, 192.168.202.119@tcp119 [12338.089771] LNetError: 468948:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12338.101688] LNetError: 468948:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 17 previous similar messages [12338.116619] LNetError: Protocol error connecting to 192.168.202.119@tcp123 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12338.133946] LNetError: Skipped 6 previous similar messages [12339.826584] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12342.181853] LNetError: 471512:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12342.198382] LNetError: Protocol error connecting to 192.168.202.119@tcp123 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12350.294934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12350.346530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12350.369092] LNetError: 471510:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12350.374508] LNetError: Protocol error connecting to 192.168.202.119@tcp123 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12350.457516] Lustre: 471508:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12350.559658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12350.562530] Lustre: Skipped 1 previous similar message [12350.604055] Lustre: lustre-MDT0001: new disk, initializing [12350.688678] Lustre: 472294:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [12350.704253] Lustre: 472294:0:(lustre_peer.c:105:class_add_uuid()) Skipped 12 previous similar messages [12350.861153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12350.949194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12350.956029] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12355.045288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12362.348197] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12365.728342] LNetError: 471511:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12365.741531] LNetError: Protocol error connecting to 192.168.202.119@tcp124 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12366.228625] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12372.637275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12372.721475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12373.053809] Lustre: lustre-OST0000: new disk, initializing [12373.056612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12373.153747] Lustre: 473265:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [12373.159101] Lustre: 473265:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12373.195629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12373.930369] LNetError: 471516:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12373.941312] LNetError: 471516:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [12373.945220] LNetError: Protocol error connecting to 192.168.202.119@tcp115 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12373.949528] LNetError: Skipped 8 previous similar messages [12378.382919] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12380.185260] Lustre: 473681:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.119@tcp: can't add NID 192.168.202.119@tcp130: rc = -75 [12380.189428] Lustre: 473681:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12380.260970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12380.271840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12380.473175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12386.007057] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12391.330267] LNetError: 471512:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12391.338806] LNetError: 471512:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 13 previous similar messages [12391.348146] LNetError: Protocol error connecting to 192.168.202.119@tcp124 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12391.354804] LNetError: Skipped 13 previous similar messages [12396.011642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12396.011898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12396.022654] LustreError: Skipped 1 previous similar message [12396.022712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12396.034695] Lustre: Skipped 1 previous similar message [12396.447955] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp123 [12399.013584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12399.016454] Lustre: Skipped 1 previous similar message [12399.180569] LustreError: 474101:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12399.265558] Lustre: server umount lustre-OST0000 complete [12404.194400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12404.212369] Lustre: Skipped 1 previous similar message [12404.216227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12404.225159] Lustre: Skipped 1 previous similar message [12408.994573] LustreError: 474303:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12409.006307] LustreError: 474303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12409.206952] Lustre: server umount lustre-MDT0000 complete [12413.594717] LustreError: 471498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344309 with bad export cookie 7418283142253273591 [12413.611741] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12413.620724] LustreError: 471498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12413.903880] LustreError: 474505:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12413.907328] LustreError: 474505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12414.287719] Lustre: server umount lustre-MDT0001 complete [12428.380886] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12429.282467] LNetError: 471513:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12429.298451] LNetError: 471513:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 22 previous similar messages [12429.308323] LNetError: Protocol error connecting to 192.168.202.119@tcp124 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12429.330388] LNetError: Skipped 22 previous similar messages [12431.398232] Key type lgssc unregistered [12431.730756] LNet: 475045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12431.735813] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12432.799316] LNet: Removed LNI 192.168.202.119@tcp [12432.814635] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12432.819896] LNet: Removed LNI 192.168.202.119@tcp100 [12433.887201] LNet: Removed LNI 192.168.202.119@tcp101 [12433.901881] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12433.920844] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12433.941460] LNet: Removed LNI 192.168.202.119@tcp102 [12434.986882] LNet: Removed LNI 192.168.202.119@tcp103 [12436.063868] LNet: Removed LNI 192.168.202.119@tcp104 [12436.075186] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12436.082971] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12436.086682] LNet: Removed LNI 192.168.202.119@tcp105 [12437.151678] LNet: Removed LNI 192.168.202.119@tcp106 [12438.180381] LNet: Removed LNI 192.168.202.119@tcp107 [12439.199129] LNet: Removed LNI 192.168.202.119@tcp108 [12440.224143] LNet: Removed LNI 192.168.202.119@tcp109 [12440.228935] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12440.232343] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12441.248753] LNet: Removed LNI 192.168.202.119@tcp110 [12442.271279] LNet: Removed LNI 192.168.202.119@tcp111 [12443.296280] LNet: Removed LNI 192.168.202.119@tcp112 [12444.319184] LNet: Removed LNI 192.168.202.119@tcp113 [12445.343189] LNet: Removed LNI 192.168.202.119@tcp114 [12446.368297] LNet: Removed LNI 192.168.202.119@tcp115 [12447.391795] LNet: Removed LNI 192.168.202.119@tcp116 [12448.415145] LNet: Removed LNI 192.168.202.119@tcp117 [12448.420849] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12448.423894] LNetError: 475045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [12449.439204] LNet: Removed LNI 192.168.202.119@tcp118 [12450.463228] LNet: Removed LNI 192.168.202.119@tcp119 [12451.496463] LNet: Removed LNI 192.168.202.119@tcp120 [12452.511171] LNet: Removed LNI 192.168.202.119@tcp121 [12453.540892] LNet: Removed LNI 192.168.202.119@tcp122 [12453.542634] LNet: Removed LNI 192.168.202.119@tcp123 [12454.561186] LNet: Removed LNI 192.168.202.119@tcp124 [12455.583165] LNet: Removed LNI 192.168.202.119@tcp125 [12456.617772] LNet: Removed LNI 192.168.202.119@tcp126 [12457.631159] LNet: Removed LNI 192.168.202.119@tcp127 [12458.655580] LNet: Removed LNI 192.168.202.119@tcp128 [12459.680568] LNet: Removed LNI 192.168.202.119@tcp129 [12459.689877] LNet: Removed LNI 192.168.202.119@tcp130 [12459.704811] LNet: Removed LNI 192.168.202.119@tcp131 [12459.711441] LNet: Removed LNI 192.168.202.119@tcp132 [12459.722391] LNet: Removed LNI 192.168.202.119@tcp133 [12459.724812] LNet: Removed LNI 192.168.202.119@tcp134 [12459.734766] LNet: Removed LNI 192.168.202.119@tcp135 [12460.306834] Key type .llcrypt unregistered [12460.308651] Key type ._llcrypt unregistered [12482.583788] Key type ._llcrypt registered [12482.595131] Key type .llcrypt registered [12482.686664] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [12495.158813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12495.710919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12495.764810] alg: No test for adler32 (adler32-zlib) [12496.779516] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [12496.977392] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12498.583196] Key type lgssc registered [12499.565505] Lustre: Echo OBD driver; http://www.lustre.org/ [12507.645279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12513.397700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12518.573383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12524.658532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12531.830426] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:53:45 (1763344425) [12546.402808] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12548.727701] Key type lgssc unregistered [12549.056159] LNet: 479088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12549.081698] LNetError: 479088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12549.099531] LNet: Removed LNI 192.168.202.119@tcp [12550.017721] Key type .llcrypt unregistered [12550.019416] Key type ._llcrypt unregistered [12566.545048] Key type ._llcrypt registered [12566.546519] Key type .llcrypt registered [12566.677505] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12568.152326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12568.186704] alg: No test for adler32 (adler32-zlib) [12569.251359] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [12569.401736] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12571.079181] Key type lgssc registered [12572.279458] Lustre: Echo OBD driver; http://www.lustre.org/ [12580.174332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12580.222963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12580.231739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.438679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12581.490325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12581.565634] Lustre: lustre-MDT0000: new disk, initializing [12581.667386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12581.685093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12584.531527] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12594.061340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12594.144497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12594.269501] Lustre: 480280:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12594.355787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12594.365879] Lustre: Skipped 1 previous similar message [12594.474495] Lustre: lustre-MDT0001: new disk, initializing [12594.588758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12594.674620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12594.678117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12599.549673] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12608.001904] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12611.891080] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12617.888576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12617.967427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12618.189421] Lustre: lustre-OST0000: new disk, initializing [12618.198412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12618.302428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12622.825463] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12627.017886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12627.032699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12627.155393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12628.706700] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12631.252583] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing lnet_if_list [12634.572057] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12634.591767] LNet: Added LNI 192.168.202.119@tcp10 [8/256/0/180] [12638.439952] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12638.466602] LNet: Added LNI 192.168.202.119@tcp11 [8/256/0/180] [12641.694288] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12641.723985] LNet: Added LNI 192.168.202.119@tcp12 [8/256/0/180] [12645.487505] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12645.510059] LNet: Added LNI 192.168.202.119@tcp13 [8/256/0/180] [12648.687133] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12648.715328] LNet: Added LNI 192.168.202.119@tcp14 [8/256/0/180] [12652.219456] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12652.237879] LNet: Added LNI 192.168.202.119@tcp15 [8/256/0/180] [12655.987821] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12656.025806] LNet: Added LNI 192.168.202.119@tcp16 [8/256/0/180] [12660.116502] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12660.155269] LNet: Added LNI 192.168.202.119@tcp17 [8/256/0/180] [12664.689528] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12664.728128] LNet: Added LNI 192.168.202.119@tcp18 [8/256/0/180] [12668.912991] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12668.943294] LNet: Added LNI 192.168.202.119@tcp19 [8/256/0/180] [12672.751524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12672.777024] LNet: Added LNI 192.168.202.119@tcp20 [8/256/0/180] [12676.998257] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12677.023407] LNet: Added LNI 192.168.202.119@tcp21 [8/256/0/180] [12681.188524] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12681.222253] LNet: Added LNI 192.168.202.119@tcp22 [8/256/0/180] [12685.456885] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12685.479671] LNet: Added LNI 192.168.202.119@tcp23 [8/256/0/180] [12690.302642] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12690.344308] LNet: Added LNI 192.168.202.119@tcp24 [8/256/0/180] [12695.000427] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12695.025422] LNet: Added LNI 192.168.202.119@tcp25 [8/256/0/180] [12700.236996] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12700.253918] LNet: Added LNI 192.168.202.119@tcp26 [8/256/0/180] [12705.162427] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12705.185575] LNet: Added LNI 192.168.202.119@tcp27 [8/256/0/180] [12705.213781] LNetError: 479539:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12705.224991] LNetError: Protocol error connecting to 192.168.202.119@tcp26 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12709.993435] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12710.013226] LNet: Added LNI 192.168.202.119@tcp28 [8/256/0/180] [12710.022227] LNetError: 479541:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12710.035761] LNetError: Protocol error connecting to 192.168.202.119@tcp26 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12715.155853] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12715.188247] LNet: Added LNI 192.168.202.119@tcp29 [8/256/0/180] [12715.197737] LNetError: 479540:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12715.210626] LNetError: Protocol error connecting to 192.168.202.119@tcp28 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12715.872442] LNet: 1 local NIs in recovery (showing 1): 192.168.202.119@tcp28 [12718.948552] LNetError: 479542:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12718.957952] LNetError: 479542:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [12718.976287] LNetError: Protocol error connecting to 192.168.202.119@tcp28 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12718.980148] LNetError: Skipped 1 previous similar message [12721.108710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12721.130483] LNet: Added LNI 192.168.202.119@tcp30 [8/256/0/180] [12726.535720] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12726.566180] LNet: Added LNI 192.168.202.119@tcp31 [8/256/0/180] [12726.575881] LNetError: 479540:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12726.594228] LNetError: 479540:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [12726.599642] LNetError: Protocol error connecting to 192.168.202.119@tcp29 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12726.614962] LNetError: Skipped 1 previous similar message [12731.876546] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12731.896402] LNet: Added LNI 192.168.202.119@tcp32 [8/256/0/180] [12736.400201] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12736.421789] LNet: Added LNI 192.168.202.119@tcp33 [8/256/0/180] [12736.429800] LNetError: 479540:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12736.443900] LNetError: 479540:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 11 previous similar messages [12736.454121] LNetError: Protocol error connecting to 192.168.202.119@tcp32 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12736.460171] LNetError: Skipped 11 previous similar messages [12740.266248] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12740.294632] LNet: Added LNI 192.168.202.119@tcp34 [8/256/0/180] [12744.718639] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12744.738842] LNet: Added LNI 192.168.202.119@tcp35 [8/256/0/180] [12748.721539] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12748.737028] LNet: Added LNI 192.168.202.119@tcp36 [8/256/0/180] [12752.330469] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12752.350816] LNet: Added LNI 192.168.202.119@tcp37 [8/256/0/180] [12753.764146] LNetError: 479542:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12753.769219] LNetError: 479542:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 17 previous similar messages [12753.772943] LNetError: Protocol error connecting to 192.168.202.119@tcp35 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12753.777244] LNetError: Skipped 19 previous similar messages [12757.005508] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12757.032241] LNet: Added LNI 192.168.202.119@tcp38 [8/256/0/180] [12762.107566] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12762.128482] LNet: Added LNI 192.168.202.119@tcp39 [8/256/0/180] [12767.992489] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12768.019784] LNet: Added LNI 192.168.202.119@tcp40 [8/256/0/180] [12773.866620] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12773.899527] LNet: Added LNI 192.168.202.119@tcp41 [8/256/0/180] [12776.351414] LNet: 10 local NIs in recovery (showing 10): 192.168.202.119@tcp34, 192.168.202.119@tcp39, 192.168.202.119@tcp38, 192.168.202.119@tcp35, 192.168.202.119@tcp36, 192.168.202.119@tcp32, 192.168.202.119@tcp30, 192.168.202.119@tcp29, 192.168.202.119@tcp28, 192.168.202.119@tcp40 [12780.455126] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12780.499878] LNet: Added LNI 192.168.202.119@tcp42 [8/256/0/180] [12787.301347] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12787.327987] LNet: Added LNI 192.168.202.119@tcp43 [8/256/0/180] [12787.346700] LNetError: 479541:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12787.351041] LNetError: 479541:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 40 previous similar messages [12787.372877] LNetError: Protocol error connecting to 192.168.202.119@tcp42 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12787.385648] LNetError: Skipped 38 previous similar messages [12794.403576] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12794.436455] LNet: Added LNI 192.168.202.119@tcp44 [8/256/0/180] [12800.324444] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12800.345951] LNet: Added LNI 192.168.202.119@tcp45 [8/256/0/180] [12807.017982] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12807.045937] LNet: Added LNI 192.168.202.119@tcp46 [8/256/0/180] [12813.999961] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12814.026125] LNet: Added LNI 192.168.202.119@tcp47 [8/256/0/180] [12822.614267] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12822.649916] LNet: Added LNI 192.168.202.119@tcp48 [8/256/0/180] [12830.696477] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12830.719803] LNet: Added LNI 192.168.202.119@tcp49 [8/256/0/180] [12837.908905] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12837.935719] LNet: Added LNI 192.168.202.119@tcp50 [8/256/0/180] [12852.256650] LNetError: 479542:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.119 [12852.279309] LNetError: 479542:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 74 previous similar messages [12852.301167] LNetError: Protocol error connecting to 192.168.202.119@tcp49 at host 192.168.202.119:988: is it running a compatible version of Lustre? [12852.341931] LNetError: Skipped 74 previous similar messages [12857.826779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12857.829956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12857.835173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12857.849942] Lustre: Skipped 1 previous similar message [12857.856567] Lustre: Skipped 1 previous similar message [12861.509218] LustreError: 504546:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12861.660920] Lustre: server umount lustre-OST0000 complete [12866.016514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12866.018995] LustreError: Skipped 1 previous similar message [12866.033760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12866.053379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12868.065626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12868.069738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12868.093300] Lustre: Skipped 1 previous similar message [12871.548674] LustreError: 504747:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12871.551790] LustreError: 504747:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12871.696461] Lustre: server umount lustre-MDT0000 complete [12874.399471] LustreError: 480271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344769 with bad export cookie 12426603780988535170 [12874.406139] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12874.512708] LustreError: 504948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12874.515736] LustreError: 504948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12874.638069] Lustre: server umount lustre-MDT0001 complete [12883.807504] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [12886.148552] Key type lgssc unregistered [12886.520473] LNet: 505488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12886.525594] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12886.527819] LNet: Removed LNI 192.168.202.119@tcp [12887.583186] LNet: Removed LNI 192.168.202.119@tcp10 [12887.584713] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12887.596082] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12887.609240] LNet: Removed LNI 192.168.202.119@tcp11 [12888.671178] LNet: Removed LNI 192.168.202.119@tcp12 [12888.673203] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12888.684237] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12888.689692] LNet: Removed LNI 192.168.202.119@tcp13 [12889.695215] LNet: Removed LNI 192.168.202.119@tcp14 [12890.719875] LNet: Removed LNI 192.168.202.119@tcp15 [12890.726684] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12890.735991] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12890.742893] LNet: Removed LNI 192.168.202.119@tcp16 [12891.807991] LNet: Removed LNI 192.168.202.119@tcp17 [12892.831196] LNet: Removed LNI 192.168.202.119@tcp18 [12893.855628] LNet: Removed LNI 192.168.202.119@tcp19 [12894.879209] LNet: Removed LNI 192.168.202.119@tcp20 [12894.886271] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12894.891034] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12895.903364] LNet: Removed LNI 192.168.202.119@tcp21 [12896.927391] LNet: Removed LNI 192.168.202.119@tcp22 [12897.951217] LNet: Removed LNI 192.168.202.119@tcp23 [12898.975163] LNet: Removed LNI 192.168.202.119@tcp24 [12899.999595] LNet: Removed LNI 192.168.202.119@tcp25 [12901.023600] LNet: Removed LNI 192.168.202.119@tcp26 [12901.032707] LNet: Removed LNI 192.168.202.119@tcp27 [12901.050696] LNet: Removed LNI 192.168.202.119@tcp28 [12901.067481] LNet: Removed LNI 192.168.202.119@tcp29 [12901.069424] LNet: Removed LNI 192.168.202.119@tcp30 [12902.111182] LNet: Removed LNI 192.168.202.119@tcp31 [12903.136574] LNet: Removed LNI 192.168.202.119@tcp32 [12903.139488] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12903.143982] LNetError: 505488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 11 previous similar messages [12903.147728] LNet: Removed LNI 192.168.202.119@tcp33 [12904.159181] LNet: Removed LNI 192.168.202.119@tcp34 [12904.161074] LNet: Removed LNI 192.168.202.119@tcp35 [12904.166242] LNet: Removed LNI 192.168.202.119@tcp36 [12905.183241] LNet: Removed LNI 192.168.202.119@tcp37 [12905.185125] LNet: Removed LNI 192.168.202.119@tcp38 [12905.193250] LNet: Removed LNI 192.168.202.119@tcp39 [12905.197518] LNet: Removed LNI 192.168.202.119@tcp40 [12905.203219] LNet: Removed LNI 192.168.202.119@tcp41 [12905.213090] LNet: Removed LNI 192.168.202.119@tcp42 [12905.225526] LNet: Removed LNI 192.168.202.119@tcp43 [12905.233198] LNet: Removed LNI 192.168.202.119@tcp44 [12905.237615] LNet: Removed LNI 192.168.202.119@tcp45 [12906.271377] LNet: Removed LNI 192.168.202.119@tcp46 [12906.275825] LNet: Removed LNI 192.168.202.119@tcp47 [12906.277443] LNet: Removed LNI 192.168.202.119@tcp48 [12906.296149] LNet: Removed LNI 192.168.202.119@tcp49 [12906.308727] LNet: Removed LNI 192.168.202.119@tcp50 [12906.991144] Key type .llcrypt unregistered [12906.992647] Key type ._llcrypt unregistered [12925.847824] Key type ._llcrypt registered [12925.849366] Key type .llcrypt registered [12925.913564] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [12926.435526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12926.485733] alg: No test for adler32 (adler32-zlib) [12927.500801] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [12927.735281] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [12929.407185] Key type lgssc registered [12930.607611] Lustre: Echo OBD driver; http://www.lustre.org/ [12937.631614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12937.655722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12939.147517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12942.669367] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12949.487627] LustreError: 506735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12949.509059] LustreError: 506735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12951.042586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12951.588194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12955.580392] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12961.163452] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12964.493760] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12972.159640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12972.710264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12979.126774] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [12985.390648] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12995.772118] loop0: detected capacity change from 0 to 204800000 [12995.971979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12995.980291] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12995.985636] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12996.028196] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13002.113212] loop0: detected capacity change from 0 to 204800000 [13002.346424] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13002.372217] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13002.404484] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13002.610699] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13008.312710] loop0: detected capacity change from 0 to 204800000 [13013.596304] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13013.677693] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13013.778825] Lustre: fs15246-OST0000: new disk, initializing [13013.781276] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13013.848744] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13019.965813] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13029.580185] loop1: detected capacity change from 0 to 204800000 [13035.201715] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13035.261915] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13035.329950] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13035.391777] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13035.477381] Lustre: fs15246-MDT0000: new disk, initializing [13035.620892] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13035.656701] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13035.735771] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13036.771169] Lustre: 511125:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000bf8b8e3a x1849001061166592/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763344948 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13039.764363] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13045.861063] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13045.867526] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13045.896990] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13064.175774] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.185499] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13067.627436] LustreError: 512249:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [13067.712755] Lustre: server umount fs15246-MDT0000 complete [13069.287709] LustreError: 506732:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13074.404460] LustreError: 506732:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13077.260198] LustreError: 512751:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [13077.262519] LustreError: 512751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13077.356552] Lustre: server umount fs15246-OST0000 complete [13088.741300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13088.762720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13088.785558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13093.316819] LustreError: 513253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13093.333449] LustreError: 513253:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [13093.521397] Lustre: server umount lustre-OST0000 complete [13100.000070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13100.009534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.034477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13100.041358] Lustre: Skipped 2 previous similar messages [13100.049355] Lustre: Skipped 3 previous similar messages [13103.015321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13103.518564] LustreError: 513456:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [13103.524854] LustreError: 513456:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13103.696036] Lustre: server umount lustre-MDT0000 complete [13107.910534] LustreError: 506715:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345003 with bad export cookie 15728447879573802502 [13107.916515] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13108.096833] LustreError: 513658:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [13108.114672] LustreError: 513658:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13108.273956] Lustre: server umount lustre-MDT0001 complete [13119.490323] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13122.387707] Key type lgssc unregistered [13122.658896] LNet: 514190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13122.672973] LNetError: 514190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13123.693371] LNet: Removed LNI 192.168.202.119@tcp [13124.351702] Key type .llcrypt unregistered [13124.355680] Key type ._llcrypt unregistered [13140.149260] Key type ._llcrypt registered [13140.154285] Key type .llcrypt registered [13140.261688] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13141.164432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13141.181352] alg: No test for adler32 (adler32-zlib) [13142.218467] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [13142.416708] LNet: Added LNI 192.168.202.119@tcp [8/256/0/180] [13144.073564] Key type lgssc registered [13144.956183] Lustre: Echo OBD driver; http://www.lustre.org/ [13151.161771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13156.059517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13160.912546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13165.264229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13178.276862] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13187.683920] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13196.054153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13201.306842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13207.062354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13213.372741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13219.966634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13220.052524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13220.077636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13221.356762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13221.391058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13221.487596] Lustre: lustre-MDT0000: new disk, initializing [13221.605836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13221.625909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13225.385710] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13236.095382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13236.165400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13236.248198] Lustre: 518624:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13236.315515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13236.317602] Lustre: Skipped 1 previous similar message [13236.469786] Lustre: lustre-MDT0001: new disk, initializing [13236.638517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13236.719919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13236.742807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13241.055401] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13248.381409] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13251.619695] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13258.267501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13258.368725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13258.655659] Lustre: lustre-OST0000: new disk, initializing [13258.669773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13258.776172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13264.102553] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_default_debug -1 all [13268.558709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13268.571965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13268.679583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13271.563896] Lustre: DEBUG MARKER: oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13274.936499] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13275.142480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13278.794116] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13278.990989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13284.320990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13284.327603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13284.333208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13284.336140] Lustre: Skipped 1 previous similar message [13284.353033] Lustre: Skipped 1 previous similar message [13289.441441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13289.447264] Lustre: Skipped 1 previous similar message [13294.588493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13295.071160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13295.124514] LustreError: 521565:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [13295.290488] Lustre: server umount lustre-OST0000 complete [13299.687304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.693706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13299.694307] Lustre: Skipped 1 previous similar message [13299.700125] Lustre: Skipped 2 previous similar messages [13304.801032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13304.817496] Lustre: Skipped 1 previous similar message [13304.848320] LustreError: 521768:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13304.851565] LustreError: 521768:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13304.934437] Lustre: server umount lustre-MDT0000 complete [13308.306818] LustreError: 518618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345203 with bad export cookie 5063029145159835113 [13308.307599] LustreError: MGC192.168.202.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13308.329700] LustreError: 518618:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13308.471485] LustreError: 521970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13308.474355] LustreError: 521970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13308.754965] Lustre: server umount lustre-MDT0001 complete [13331.889795] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing set_hostid [13341.490324] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing load_modules_local [13351.883792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13358.304099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13364.190768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13369.676561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13371.079797] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13078 sec ========= 21:07:44 (1763345264) [13372.648157] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:07:46 (1763345266) === [13376.213872] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:07:49 (1763345269) === [13410.754401] Lustre: DEBUG MARKER: oleg219-server.virtnet: executing unload_modules_local [13413.586961] Key type lgssc unregistered [13413.967963] LNet: 526829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13413.979338] LNetError: 526829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13413.999882] LNet: Removed LNI 192.168.202.119@tcp [13415.044322] Key type .llcrypt unregistered [13415.045834] Key type ._llcrypt unregistered