[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 503609261 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003127] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012053] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.020788] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025398] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] *** VALIDATE tmpfs *** [ 0.030457] *** VALIDATE proc *** [ 0.031256] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035069] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037028] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041032] debug: unmapping init [mem 0xffffffff86a59000-0xffffffff86a60fff] [ 0.043859] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048017] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.053204] rcu: Hierarchical SRCU implementation. [ 0.055342] smp: Bringing up secondary CPUs ... [ 0.056613] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.061035] smp: Brought up 1 node, 4 CPUs [ 0.062992] smpboot: Max logical packages: 1 [ 0.063016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.185368] node 0 deferred pages initialised in 119ms [ 0.188160] devtmpfs: initialized [ 0.189786] x86/mm: Memory block size: 128MB [ 0.195000] gcov: version magic: 0x41383552 [ 0.199516] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.205124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.207423] pinctrl core: initialized pinctrl subsystem [ 0.210220] [ 0.210754] ************************************************************* [ 0.213012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214012] ** ** [ 0.216013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.219014] ** ** [ 0.221011] ** This means that this kernel is built to expose internal ** [ 0.223015] ** IOMMU data structures, which may compromise security on ** [ 0.225012] ** your system. ** [ 0.227014] ** ** [ 0.230016] ** If you see this message and you are not debugging the ** [ 0.232013] ** kernel, report this immediately to your vendor! ** [ 0.235016] ** ** [ 0.237012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239018] ************************************************************* [ 0.242886] NET: Registered protocol family 16 [ 0.244484] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.248071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.251084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.255137] cpuidle: using governor menu [ 0.256831] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257543] PCI: Using configuration type 1 for base access [ 0.258119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.263213] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.270036] cryptd: max_cpu_qlen set to 1000 [ 0.274320] ACPI: Added _OSI(Module Device) [ 0.275018] ACPI: Added _OSI(Processor Device) [ 0.277013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.280016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.284997] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.291742] ACPI: Interpreter enabled [ 0.294095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296014] ACPI: Using IOAPIC for interrupt routing [ 0.298170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.302672] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.315343] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.318049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.321022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.325109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.330551] acpiphp: Slot [2] registered [ 0.332125] acpiphp: Slot [5] registered [ 0.333188] acpiphp: Slot [6] registered [ 0.335111] acpiphp: Slot [7] registered [ 0.336131] acpiphp: Slot [8] registered [ 0.338172] acpiphp: Slot [9] registered [ 0.340195] acpiphp: Slot [10] registered [ 0.341116] acpiphp: Slot [3] registered [ 0.343148] acpiphp: Slot [4] registered [ 0.345100] acpiphp: Slot [11] registered [ 0.346168] acpiphp: Slot [12] registered [ 0.348102] acpiphp: Slot [13] registered [ 0.350154] acpiphp: Slot [14] registered [ 0.352171] acpiphp: Slot [15] registered [ 0.353099] acpiphp: Slot [16] registered [ 0.354128] acpiphp: Slot [17] registered [ 0.356130] acpiphp: Slot [18] registered [ 0.358108] acpiphp: Slot [19] registered [ 0.359111] acpiphp: Slot [20] registered [ 0.361108] acpiphp: Slot [21] registered [ 0.362101] acpiphp: Slot [22] registered [ 0.364183] acpiphp: Slot [23] registered [ 0.365099] acpiphp: Slot [24] registered [ 0.367105] acpiphp: Slot [25] registered [ 0.369121] acpiphp: Slot [26] registered [ 0.371140] acpiphp: Slot [27] registered [ 0.373123] acpiphp: Slot [28] registered [ 0.374167] acpiphp: Slot [29] registered [ 0.376131] acpiphp: Slot [30] registered [ 0.378119] acpiphp: Slot [31] registered [ 0.380084] PCI host bridge to bus 0000:00 [ 0.382029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.390045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.393027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.397028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.401031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.403341] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.407673] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.409353] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.420999] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.426043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.429018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.436021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.439941] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.442773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.447049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.450905] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.457015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.471026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.477962] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.497022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.510031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.532023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.542238] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.547015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.574032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.588077] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.602014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.616015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.639023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.647446] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.654019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.670016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.709014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.722000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.731015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.741016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.767016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.782115] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.791014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.800016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.833014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.854582] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.856403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.860452] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.863396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.867257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.873109] iommu: Default domain type: Passthrough [ 0.875435] SCSI subsystem initialized [ 0.876143] ACPI: bus type USB registered [ 0.878186] usbcore: registered new interface driver usbfs [ 0.881080] usbcore: registered new interface driver hub [ 0.883095] usbcore: registered new device driver usb [ 0.885212] pps_core: LinuxPPS API ver. 1 registered [ 0.887010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.890057] PTP clock support registered [ 0.892972] EDAC MC: Ver: 3.0.0 [ 0.895223] PCI: Using ACPI for IRQ routing [ 0.896978] NetLabel: Initializing [ 0.899117] NetLabel: domain hash size = 128 [ 0.900010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.902074] NetLabel: unlabeled traffic allowed by default [ 0.906088] vgaarb: loaded [ 0.908142] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.910010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.917724] clocksource: Switched to clocksource kvm-clock [ 1.025291] VFS: Disk quotas dquot_6.6.0 [ 1.026765] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.029614] *** VALIDATE ramfs *** [ 1.030750] *** VALIDATE hugetlbfs *** [ 1.032453] pnp: PnP ACPI init [ 1.035081] pnp: PnP ACPI: found 6 devices [ 1.052136] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.055762] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.058397] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.060820] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.063193] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.065560] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.068484] NET: Registered protocol family 2 [ 1.071879] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.077849] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.081731] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.087453] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.090597] TCP: Hash tables configured (established 65536 bind 65536) [ 1.093980] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.098470] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.101624] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.105131] NET: Registered protocol family 1 [ 1.108313] RPC: Registered named UNIX socket transport module. [ 1.110335] RPC: Registered udp transport module. [ 1.113121] RPC: Registered tcp transport module. [ 1.114941] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.118191] NET: Registered protocol family 44 [ 1.120206] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.122967] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.125202] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.127972] PCI: CLS 0 bytes, default 64 [ 1.129460] Unpacking initramfs... [ 2.632455] debug: unmapping init [mem 0xffff907dbcc54000-0xffff907dbffbffff] [ 2.636715] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.639316] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.642467] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.152586] Initialise system trusted keyrings [ 3.154385] Key type blacklist registered [ 3.156394] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.166827] zbud: loaded [ 3.169853] *** VALIDATE nfs *** [ 3.170958] *** VALIDATE nfs4 *** [ 3.172247] pstore: using deflate compression [ 3.175475] Platform Keyring initialized [ 3.285596] NET: Registered protocol family 38 [ 3.287593] Key type asymmetric registered [ 3.289370] Asymmetric key parser 'x509' registered [ 3.292160] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.297284] io scheduler mq-deadline registered [ 3.299073] io scheduler kyber registered [ 3.301214] io scheduler bfq registered [ 3.303836] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.310986] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.313870] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.316817] ACPI: Power Button [PWRF] [ 3.325795] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.333521] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.350711] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.360544] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.389740] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.419990] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.447264] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.452346] Non-volatile memory driver v1.3 [ 3.454299] Linux agpgart interface v0.103 [ 3.491985] virtio_blk virtio1: [vda] 146872 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.496140] vda: detected capacity change from 0 to 75198464 [ 3.514632] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.519930] vdb: detected capacity change from 0 to 1073741824 [ 3.541843] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.545795] vdc: detected capacity change from 0 to 2621440000 [ 3.566223] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.569419] vdd: detected capacity change from 0 to 2621440000 [ 3.600178] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.604039] vde: detected capacity change from 0 to 4294967296 [ 3.618718] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.622324] vdf: detected capacity change from 0 to 4294967296 [ 3.631771] libphy: Fixed MDIO Bus: probed [ 3.637709] usbcore: registered new interface driver usbserial_generic [ 3.640592] usbserial: USB Serial support registered for generic [ 3.643406] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.648247] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.650947] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.654476] mousedev: PS/2 mouse device common for all mice [ 3.657810] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.661308] rtc_cmos 00:05: RTC can wake from S4 [ 3.669803] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.671398] rtc_cmos 00:05: registered as rtc0 [ 3.671449] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.671497] intel_pstate: CPU model not supported [ 3.679249] hid: raw HID events driver (C) Jiri Kosina [ 3.683858] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.686582] usbcore: registered new interface driver usbhid [ 3.696560] usbhid: USB HID core driver [ 3.698230] drop_monitor: Initializing network drop monitor service [ 3.700807] Initializing XFRM netlink socket [ 3.702990] NET: Registered protocol family 10 [ 3.706346] Segment Routing with IPv6 [ 3.707915] NET: Registered protocol family 17 [ 3.710151] mpls_gso: MPLS GSO support [ 3.715753] RAS: Correctable Errors collector initialized. [ 3.717991] AVX version of gcm_enc/dec engaged. [ 3.719856] AES CTR mode by8 optimization enabled [ 3.812368] sched_clock: Marking stable (3812337379, 0)->(4780275584, -967938205) [ 3.818835] registered taskstats version 1 [ 3.821148] Loading compiled-in X.509 certificates [ 3.823138] zswap: loaded using pool lzo/zbud [ 3.846970] Key type big_key registered [ 3.859558] Key type encrypted registered [ 3.861150] ima: No TPM chip found, activating TPM-bypass! [ 3.862921] ima: Allocated hash algorithm: sha1 [ 3.863970] ima: No architecture policies found [ 3.865063] evm: Initialising EVM extended attributes: [ 3.866225] evm: security.selinux [ 3.867154] evm: security.ima [ 3.868200] evm: security.capability [ 3.869162] evm: HMAC attrs: 0x1 [ 3.871802] rtc_cmos 00:05: setting system clock to 2026-08-27 16:12:10 UTC (1787847130) [ 3.877019] debug: unmapping init [mem 0xffffffff87a03000-0xffffffff87bfffff] [ 3.879319] debug: unmapping init [mem 0xffffffff86782000-0xffffffff86a58fff] [ 3.888073] Write protecting the kernel read-only data: 28672k [ 3.891332] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.894381] debug: unmapping init [mem 0xffffffff85714000-0xffffffff857fffff] [ 3.928759] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.936821] systemd[1]: Detected virtualization kvm. [ 3.938576] systemd[1]: Detected architecture x86-64. [ 3.940370] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.967547] systemd[1]: No hostname configured. [ 3.969564] systemd[1]: Set hostname to . [ 3.971736] random: systemd: uninitialized urandom read (16 bytes read) [ 3.974174] systemd[1]: Initializing machine ID from random generator. [ 4.101445] random: systemd: uninitialized urandom read (16 bytes read) [ 4.104329] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.109140] random: systemd: uninitialized urandom read (16 bytes read) [ 4.111927] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.116569] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.769733] device-mapper: uevent: version 1.0.3 [ 4.773196] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.510091] virtio_net virtio0 ens2: renamed from eth0 [ 5.517753] random: fast init done [ 5.631853] scsi host0: ata_piix [ 5.659059] scsi host1: ata_piix [ 5.676437] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.679191] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.286759] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.379436] random: crng init done [ 10.381440] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.829527] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.058972] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.329499] SELinux: Disabled at runtime. [ 12.393182] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.402587] systemd[1]: Detected virtualization kvm. [ 12.406559] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.929444] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.933106] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.937804] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.941249] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.945202] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.955621] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.960629] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ 13.180777] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.469830] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.787775] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.840382] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.975505] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.987326] EDAC sbridge: Ver: 1.1.2 [ 15.741589] Key type dns_resolver registered [ 16.049685] NFS: Registering the id_resolver key type [ 16.051409] Key type id_resolver registered [ 16.052942] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg642-server login: [ 45.243951] libcfs: loading out-of-tree module taints kernel. [ 45.304649] Key type ._llcrypt registered [ 45.313321] Key type .llcrypt registered [ 45.426871] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 63.480671] hrtimer: interrupt took 4661568 ns [ 66.466798] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 68.210742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.236566] alg: No test for adler32 (adler32-zlib) [ 69.936762] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 70.959588] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 72.697475] Key type lgssc registered [ 74.066607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.888759] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.087849] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 139.909354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.933317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.194357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.219589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.280388] Lustre: lustre-MDT0000: new disk, initializing [ 141.339240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.352573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.541758] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.062320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.172138] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.210681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.215308] Lustre: Skipped 1 previous similar message [ 162.316433] Lustre: lustre-MDT0001: new disk, initializing [ 162.398382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.432031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.453200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.608875] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.249842] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.347840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.576361] Lustre: lustre-OST0000: new disk, initializing [ 181.580760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.585819] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.634846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.785151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.797798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.877073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.540153] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.518478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.701321] Lustre: lustre-OST0001: new disk, initializing [ 203.708213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.716832] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.798895] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.992947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.005172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.041369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.153362] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.816407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.686778] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 240.759777] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.264121] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing yml_node [ 249.649785] Lustre: DEBUG MARKER: Client: 2.17.57.86 [ 252.291746] Lustre: DEBUG MARKER: MDS: 2.17.57.86 [ 254.646959] Lustre: DEBUG MARKER: OSS: 2.17.57.86 [ 256.192425] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 12:16:21 EDT 2026 [ 271.289462] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 272.705442] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.272420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.286571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.314918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.808504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.809336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.844693] Lustre: Skipped 3 previous similar messages [ 294.248844] Lustre: server umount lustre-MDT0000 complete [ 297.891688] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847424 with bad export cookie 13706487790059322800 [ 297.899675] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.912038] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 302.047831] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.049858] LustreError: 6514:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.059714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.068864] Lustre: Skipped 3 previous similar messages [ 302.097146] LustreError: 6514:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 307.168156] LustreError: 6512:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.168218] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.189839] Lustre: Skipped 2 previous similar messages [ 308.192439] LustreError: 9525:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.206634] LustreError: 9525:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 312.288078] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.629927] Lustre: server umount lustre-MDT0001 complete [ 317.622965] Lustre: server umount lustre-OST0000 complete [ 321.812168] Lustre: server umount lustre-OST0001 complete [ 329.284809] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 339.587436] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 376.981537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.330804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.378804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.469929] Lustre: lustre-MDT0000: new disk, initializing [ 377.635029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.673758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 382.753670] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 395.863354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.999240] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.146147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.155477] Lustre: Skipped 1 previous similar message [ 396.309414] Lustre: lustre-MDT0001: new disk, initializing [ 396.441586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.507166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.529326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 401.329832] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 411.797531] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.211329] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 425.219595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 425.678744] Lustre: lustre-OST0000: new disk, initializing [ 425.693287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.702793] Lustre: 17038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 425.820756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.064266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 427.075017] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.205875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.493905] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 442.365633] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 446.748566] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 446.990866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.322582] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.569821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 458.212393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.217715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 458.234590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.255205] Lustre: Skipped 3 previous similar messages [ 459.843594] Lustre: server umount lustre-OST0000 complete [ 467.940917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.950174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.970039] Lustre: Skipped 1 previous similar message [ 467.978260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 470.005840] Lustre: server umount lustre-MDT0000 complete [ 473.572808] LustreError: 15947:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.839319] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847600 with bad export cookie 13706487790059324333 [ 473.845924] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.859254] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 474.246738] Lustre: server umount lustre-MDT0001 complete [ 478.771819] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:20:03 (1787847603) [ 482.027538] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 483.964834] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:20:09 (1787847609) [ 490.629664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.223607] LustreError: 19448:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.255042] LustreError: 19448:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 491.323677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.229752] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 500.713558] LustreError: 19449:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.744486] LustreError: 19449:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 506.249487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.954868] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 519.066652] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.773287] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 532.645920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.336355] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 546.834329] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 566.753275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 566.759975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.782282] Lustre: Skipped 1 previous similar message [ 566.791337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.802396] Lustre: Skipped 2 previous similar messages [ 569.254282] Lustre: server umount lustre-OST0000 complete [ 573.408349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 573.417030] LustreError: Skipped 1 previous similar message [ 579.507964] Lustre: server umount lustre-MDT0000 complete [ 583.137960] LustreError: 19448:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.159204] LustreError: 19448:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 583.181160] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847709 with bad export cookie 13706487790059325215 [ 583.187548] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 583.198789] LustreError: 19428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 583.562786] Lustre: server umount lustre-MDT0001 complete [ 594.152563] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 596.693637] Key type lgssc unregistered [ 596.945654] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.950937] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.975177] LNet: Removed LNI 192.168.206.142@tcp [ 597.917151] Key type .llcrypt unregistered [ 597.923475] Key type ._llcrypt unregistered [ 620.773626] Key type ._llcrypt registered [ 620.777685] Key type .llcrypt registered [ 620.896994] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 621.858533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 621.953155] alg: No test for adler32 (adler32-zlib) [ 622.990995] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 623.249425] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 624.896752] Key type lgssc registered [ 625.948759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.970534] loop: module loaded [ 634.030781] loop0: detected capacity change from 0 to 204800000 [ 634.253962] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.270211] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.595445] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.734798] loop0: detected capacity change from 0 to 204800000 [ 640.896624] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.914942] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 640.924820] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 646.474886] loop0: detected capacity change from 0 to 204800000 [ 652.500023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 652.540617] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.828832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 653.926198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 653.995653] Lustre: lustre-MDT0000: new disk, initializing [ 654.186282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.228659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 658.530331] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 667.034644] loop1: detected capacity change from 0 to 204800000 [ 672.729619] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 672.980946] Lustre: lustre-OST07c6: new disk, initializing [ 672.984672] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 672.990583] Lustre: Skipped 1 previous similar message [ 672.994867] Lustre: 26193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 673.075497] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 678.527494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 678.542847] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 678.587236] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 680.419494] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 686.160739] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 686.170224] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 686.175682] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 686.184740] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 686.195277] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 686.206138] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 703.959398] Lustre: 26201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 703.989769] Lustre: 26201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.003883] Lustre: 26201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.010982] Lustre: 26201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 704.019447] Lustre: 26201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 704.029764] Lustre: 26201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.044532] Lustre: 26201:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 709.053900] Lustre: 26201:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 709.058703] Lustre: 26201:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 709.069456] Lustre: 26201:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.080888] Lustre: 26201:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.092701] Lustre: 26201:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.100532] Lustre: 26201:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 709.105068] Lustre: 26201:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.109023] Lustre: 26201:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 709.115268] Lustre: 26201:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.123330] Lustre: 26201:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.126908] Lustre: 26201:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.015482] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 714.023484] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 714.029831] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 714.037694] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.044430] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.050266] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.070106] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 714.075383] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.084197] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 714.090068] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 714.096397] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 714.101984] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.680242] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 718.695187] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 718.706759] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 718.714717] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.725154] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.730623] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.740152] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 718.749225] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.754293] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.760980] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.766184] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.770811] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.751876] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 729.766386] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 729.773086] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 729.777157] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.785603] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 729.789892] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.798858] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 729.803471] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.808528] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 729.813313] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 729.818181] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 729.822147] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 779.213155] Lustre: server umount lustre-OST07c6 complete [ 797.819797] Lustre: server umount lustre-MDT0000 complete [ 814.987409] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 817.767431] Key type lgssc unregistered [ 818.092871] LNet: 28770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.103438] LNetError: 28770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.121398] LNet: Removed LNI 192.168.206.142@tcp [ 818.915875] Key type .llcrypt unregistered [ 818.921848] Key type ._llcrypt unregistered [ 838.385713] Key type ._llcrypt registered [ 838.388348] Key type .llcrypt registered [ 838.516229] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 839.679902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.796301] alg: No test for adler32 (adler32-zlib) [ 840.955561] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 841.200946] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 842.863188] Key type lgssc registered [ 844.127747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 853.967541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 854.019591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.540329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.300425] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 865.765778] LustreError: 30043:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.815543] LustreError: 30043:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 870.406751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.090748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.829449] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 884.281167] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 889.568834] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 896.441284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.742559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.867286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 903.789825] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 911.943709] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.934099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.939435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.972755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.988256] Lustre: Skipped 1 previous similar message [ 938.457686] Lustre: server umount lustre-OST0000 complete [ 947.169076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.196233] Lustre: Skipped 1 previous similar message [ 947.203747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.565933] Lustre: server umount lustre-MDT0000 complete [ 952.302014] LustreError: 31225:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.321653] LustreError: 31225:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 954.557183] LustreError: 30024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848081 with bad export cookie 6729921796795623906 [ 954.564857] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.569042] LustreError: 30024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 954.950423] Lustre: server umount lustre-MDT0001 complete [ 967.297753] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 970.720874] Key type lgssc unregistered [ 971.117547] LNet: 33785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.131328] LNetError: 33785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.152927] LNet: Removed LNI 192.168.206.142@tcp [ 972.352768] Key type .llcrypt unregistered [ 972.355128] Key type ._llcrypt unregistered [ 992.521495] Key type ._llcrypt registered [ 992.525484] Key type .llcrypt registered [ 992.630966] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 994.184982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.217135] alg: No test for adler32 (adler32-zlib) [ 995.356721] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 995.595699] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 997.312594] Key type lgssc registered [ 998.477666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1006.696368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1006.729907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.155688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.645567] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1018.340251] LustreError: 35055:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.374082] LustreError: 35055:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1021.387340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.924178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.095464] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1035.533906] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1042.106937] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1050.368769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1050.805165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.870914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1058.857658] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1067.169475] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1078.753890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.766391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.786086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.865714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.883530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1084.773503] Lustre: server umount lustre-OST0000 complete [ 1093.096537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.127040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.137435] Lustre: Skipped 2 previous similar messages [ 1095.921765] Lustre: server umount lustre-MDT0000 complete [ 1098.212166] LustreError: 35783:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.239296] LustreError: 35783:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1099.868386] LustreError: 35036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848226 with bad export cookie 8320396086248986329 [ 1099.885406] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.903284] LustreError: 35036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1100.211253] Lustre: server umount lustre-MDT0001 complete [ 1110.750389] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 1113.843228] Key type lgssc unregistered [ 1114.113840] LNet: 38596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.120526] LNetError: 38596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.175302] LNet: Removed LNI 192.168.206.142@tcp [ 1116.154511] Key type .llcrypt unregistered [ 1116.156654] Key type ._llcrypt unregistered [ 1140.418681] Key type ._llcrypt registered [ 1140.421298] Key type .llcrypt registered [ 1140.527440] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 1141.461202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.515276] alg: No test for adler32 (adler32-zlib) [ 1142.654951] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1142.879584] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 1144.607175] Key type lgssc registered [ 1145.560732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1153.554707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1153.579972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.281595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1159.456808] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1164.777964] LustreError: 40014:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.830294] LustreError: 40014:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1168.820969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.267057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.843373] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1182.554779] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1189.150611] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1196.416748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1196.733951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.809542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1203.149945] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1211.285710] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1224.234357] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.42@tcp (stopping) [ 1224.678660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.706845] Lustre: Skipped 2 previous similar messages [ 1227.619865] Lustre: server umount lustre-MDT0000 complete [ 1229.316291] LustreError: 40010:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.807673] LustreError: 39996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848358 with bad export cookie 18389319601200995980 [ 1231.822144] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.827316] LustreError: 39996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1234.432793] LustreError: 40009:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.461804] LustreError: 40009:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1234.495277] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.42@tcp (stopping) [ 1234.504717] Lustre: Skipped 3 previous similar messages [ 1234.913551] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.966170] LustreError: 40010:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.983314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1236.993746] LustreError: 40010:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1237.004163] Lustre: Skipped 1 previous similar message [ 1239.554191] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.42@tcp (stopping) [ 1242.081930] LustreError: 40011:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.130349] LustreError: 40011:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1244.686975] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.42@tcp (stopping) [ 1244.692806] Lustre: Skipped 1 previous similar message [ 1246.175234] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1246.390780] Lustre: server umount lustre-MDT0001 complete [ 1261.549273] Lustre: server umount lustre-OST0000 complete [ 1277.756957] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 1280.002213] Key type lgssc unregistered [ 1280.153668] LNet: 43882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.157214] LNetError: 43882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.166867] LNet: Removed LNI 192.168.206.142@tcp [ 1280.887571] Key type .llcrypt unregistered [ 1280.891377] Key type ._llcrypt unregistered [ 1304.437824] Key type ._llcrypt registered [ 1304.439796] Key type .llcrypt registered [ 1304.532969] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 1305.408961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.438951] alg: No test for adler32 (adler32-zlib) [ 1306.473450] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1306.697930] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 1308.360391] Key type lgssc registered [ 1309.370215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1317.354492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1317.392588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.938936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.868694] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1327.010344] LustreError: 45297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.032518] LustreError: 45297:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1329.062659] LustreError: 45296:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.012530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.246616] LustreError: 45297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.616753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1339.393543] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1347.248743] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1352.621081] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1358.194780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1358.633200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.771560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1365.244311] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1374.594685] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1382.446411] Lustre: lustre-OST0000: Not available for connect from 192.168.206.42@tcp (stopping) [ 1384.416691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1384.423839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.426385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1384.445956] Lustre: Skipped 1 previous similar message [ 1387.540169] Lustre: server umount lustre-OST0000 complete [ 1404.898665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.901467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.907195] Lustre: Skipped 1 previous similar message [ 1404.925016] Lustre: Skipped 2 previous similar messages [ 1410.971327] Lustre: server umount lustre-MDT0000 complete [ 1411.055347] LustreError: 45296:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.585888] LustreError: 45277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848541 with bad export cookie 16678317270502591434 [ 1414.597145] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.990488] Lustre: server umount lustre-MDT0001 complete [ 1432.316427] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 1434.946920] Key type lgssc unregistered [ 1435.245725] LNet: 49008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.263512] LNetError: 49008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.284823] LNet: Removed LNI 192.168.206.142@tcp [ 1436.152684] Key type .llcrypt unregistered [ 1436.155119] Key type ._llcrypt unregistered [ 1458.318799] Key type ._llcrypt registered [ 1458.320487] Key type .llcrypt registered [ 1458.414022] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 1459.656541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1459.794520] alg: No test for adler32 (adler32-zlib) [ 1460.777704] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1461.044240] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 1462.735362] Key type lgssc registered [ 1463.822109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1471.844439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1471.869915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.432706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.941589] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1482.723352] LustreError: 50427:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.755326] LustreError: 50427:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1486.244940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.553933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1491.221227] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1500.078379] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1505.851644] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1512.987320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.331516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.429402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1519.113000] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1526.686679] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1536.076782] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1539.108741] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1551.466321] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2712h37m56s [ 1553.221608] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1557.476670] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.478429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1557.489215] Lustre: Skipped 1 previous similar message [ 1558.577290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.588600] Lustre: Skipped 4 previous similar messages [ 1560.090134] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.42@tcp (stopping) [ 1561.559952] Lustre: server umount lustre-MDT0000 complete [ 1564.649492] LustreError: 50428:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.676516] LustreError: 50428:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1564.735038] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1567.249911] LustreError: 50422:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.788419] LustreError: 50422:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.813897] LustreError: 50422:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1571.374511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.479934] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.724203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.796472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1572.783802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1576.463347] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1580.118323] Lustre: DEBUG MARKER: Wait for df (36718) ... [ 1582.100248] Lustre: DEBUG MARKER: done [ 1587.177870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.188182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.212177] Lustre: Skipped 1 previous similar message [ 1587.222937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.228270] Lustre: Skipped 1 previous similar message [ 1588.196764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.584588] Lustre: server umount lustre-OST0000 complete [ 1598.434652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.436389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1598.464926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1598.464934] Lustre: Skipped 2 previous similar messages [ 1602.778277] Lustre: server umount lustre-MDT0000 complete [ 1605.608169] LustreError: 50424:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.629818] LustreError: 50424:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1606.839249] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848733 with bad export cookie 14817426224617875349 [ 1606.854800] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.863544] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1607.493712] Lustre: server umount lustre-MDT0001 complete [ 1620.101729] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 1623.002075] Key type lgssc unregistered [ 1623.312209] LNet: 55356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.332680] LNetError: 55356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.356685] LNet: Removed LNI 192.168.206.142@tcp [ 1624.416163] Key type .llcrypt unregistered [ 1624.418205] Key type ._llcrypt unregistered [ 1668.596888] Key type ._llcrypt registered [ 1668.598882] Key type .llcrypt registered [ 1668.751946] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 1670.467344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1670.523904] alg: No test for adler32 (adler32-zlib) [ 1671.766272] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1672.181092] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 1673.946215] Key type lgssc registered [ 1675.174501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1682.959875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1682.992297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.290678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1684.347566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1684.521704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.893363] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1697.927458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.071753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1698.115807] Lustre: 57566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1698.431677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1702.809410] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1711.613916] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1717.381856] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1725.352880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1725.624895] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1725.955124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.850873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1735.042382] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1743.620920] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1750.828641] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2712h41m15s [ 1752.718873] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1760.186885] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1762.737160] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1762.791576] LustreError: 59615:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1783.387929] Lustre: lustre-MDT0000: Client 76c0be6c-43b8-4c72-b040-90c97fa41db7 (at 192.168.206.42@tcp) reconnecting [ 1802.839217] LustreError: 59615:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1804.946730] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1807.660159] Lustre: DEBUG MARKER: done [ 1815.008749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1815.020724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.030661] LustreError: Skipped 1 previous similar message [ 1815.031901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.069363] Lustre: Skipped 1 previous similar message [ 1820.139500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1820.154301] Lustre: Skipped 2 previous similar messages [ 1825.248309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.263377] Lustre: Skipped 1 previous similar message [ 1829.343140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1829.577072] Lustre: server umount lustre-OST0000 complete [ 1835.494269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.500848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1835.516078] Lustre: Skipped 1 previous similar message [ 1835.538851] Lustre: Skipped 1 previous similar message [ 1839.809458] Lustre: server umount lustre-MDT0000 complete [ 1842.658347] LustreError: 57577:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.683020] LustreError: 57577:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1843.768722] LustreError: 58322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848970 with bad export cookie 8982930140805543474 [ 1843.779440] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.781762] LustreError: 58322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1844.115721] Lustre: server umount lustre-MDT0001 complete [ 1854.492678] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 1857.544487] Key type lgssc unregistered [ 1857.825824] LNet: 61901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.837260] LNetError: 61901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.873122] LNet: Removed LNI 192.168.206.142@tcp [ 1858.800661] Key type .llcrypt unregistered [ 1858.802671] Key type ._llcrypt unregistered [ 1903.695939] Key type ._llcrypt registered [ 1903.700531] Key type .llcrypt registered [ 1903.811673] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 1904.647672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.849151] alg: No test for adler32 (adler32-zlib) [ 1905.940622] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1906.244526] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 1907.935203] Key type lgssc registered [ 1909.068822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1912.388914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1912.410703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.735584] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1913.777965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1914.003618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.935064] Lustre: Failing over lustre-MDT0000 [ 1915.252368] Lustre: server umount lustre-MDT0000 complete [ 1921.753085] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:44:07 (1787849047) [ 1927.488940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.793409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.211729] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1940.457774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.529106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1940.550953] Lustre: 64701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1940.855456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.891443] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1952.265094] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1958.468457] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1965.348947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1965.546563] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1965.772768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.510879] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 1975.452908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1981.393299] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1988.375807] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1997.288819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.295956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.303566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.378229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.386934] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.414554] Lustre: Skipped 1 previous similar message [ 2003.426029] Lustre: server umount lustre-MDT0000 complete [ 2007.482345] LustreError: 66554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849134 with bad export cookie 6878010288681330280 [ 2007.489249] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2007.499338] LustreError: 66554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2008.018147] Lustre: server umount lustre-MDT0001 complete [ 2011.828907] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2016.514832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2024.353520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.999154] Lustre: 63621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849138/real 1787849138] req@00000000532ad98d x1874695586621440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849154 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2028.025428] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.087317] Lustre: 63621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849139/real 1787849139] req@0000000001296904 x1874695586621568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2033.119156] Lustre: 63623:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849143/real 1787849143] req@00000000260170ff x1874695586621696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849159 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2033.542792] LustreError: 66548:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2033.568945] LustreError: 66548:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2033.672567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.747730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2037.666682] LustreError: 66549:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.346616] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2038.691779] LustreError: 68474:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.714172] LustreError: 68474:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2039.775925] Lustre: 63622:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849149/real 1787849149] req@00000000c074e7b5 x1874695586622080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2039.799510] Lustre: 63622:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2043.895323] LustreError: 68474:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.930726] LustreError: 68474:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2044.895232] Lustre: 63622:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849154/real 1787849154] req@00000000526704f7 x1874695586622208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2047.208065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.600682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.878031] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2053.109853] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2057.292881] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2061.356286] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2067.537166] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2075.211953] Lustre: server umount lustre-MDT0000 complete [ 2077.673954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.678289] LustreError: 66548:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.693391] Lustre: Skipped 1 previous similar message [ 2077.722076] LustreError: 66548:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2079.616555] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849206 with bad export cookie 6878010288681332555 [ 2079.651082] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2079.659820] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2080.364767] Lustre: server umount lustre-MDT0001 complete [ 2087.825758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2097.087589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.167707] Lustre: 63623:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849209/real 1787849209] req@00000000c94435a6 x1874695586649216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2099.188174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.195016] Lustre: Skipped 1 previous similar message [ 2105.377727] LustreError: 63620:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f5877835 x1874695586650368/t0(0) o250->MGC192.168.206.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.755266] LustreError: 66549:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.848658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2105.931832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2109.994934] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2118.058500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.122818] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2122.287681] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2123.762753] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2130.954203] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2137.911448] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2149.355153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.363061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.364663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2149.377798] Lustre: Skipped 1 previous similar message [ 2149.430603] Lustre: Skipped 4 previous similar messages [ 2154.336762] Lustre: server umount lustre-MDT0000 complete [ 2154.476248] LustreError: 71153:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.506267] LustreError: 71153:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 2158.409016] LustreError: 64693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849285 with bad export cookie 6878010288681333934 [ 2158.417090] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2158.434408] LustreError: 64693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2158.814144] Lustre: server umount lustre-MDT0001 complete [ 2165.670627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2169.128434] Lustre: DEBUG MARKER: files compared the same [ 2180.700552] Lustre: server umount lustre-OST0000 complete [ 2195.898925] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 2198.469839] Key type lgssc unregistered [ 2198.808333] LNet: 74434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.819427] LNetError: 74434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.840637] LNet: Removed LNI 192.168.206.142@tcp [ 2199.604303] Key type .llcrypt unregistered [ 2199.606744] Key type ._llcrypt unregistered [ 2222.655488] Key type ._llcrypt registered [ 2222.666203] Key type .llcrypt registered [ 2222.786529] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 2223.659924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.691285] alg: No test for adler32 (adler32-zlib) [ 2224.858553] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2225.130134] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 2226.807217] Key type lgssc registered [ 2227.945620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2235.164919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2235.206847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.707256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2241.911063] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug +malloc all [ 2247.139363] LustreError: 75852:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.161918] LustreError: 75852:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2251.711849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.941710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2256.388791] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug +malloc all [ 2264.677137] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2270.824980] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2277.749086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2278.068335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2282.156510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2283.657306] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug +malloc all [ 2290.733780] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2302.944366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.952870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.976303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.042118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.046464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2307.048555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.713061] Lustre: server umount lustre-OST0000 complete [ 2313.696034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2313.703580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.717142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.724683] Lustre: Skipped 1 previous similar message [ 2317.283188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2317.284967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.311959] Lustre: server umount lustre-MDT0000 complete [ 2322.401037] LustreError: 75848:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.424034] LustreError: 75848:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2323.482493] LustreError: 75831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849450 with bad export cookie 17169047206925066807 [ 2323.492052] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.501135] LustreError: 75831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2324.272729] Lustre: server umount lustre-MDT0001 complete [ 2337.153679] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 2340.229744] Key type lgssc unregistered [ 2340.507647] LNet: 79388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.511770] LNetError: 79388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.538043] LNet: Removed LNI 192.168.206.142@tcp [ 2341.418734] Key type .llcrypt unregistered [ 2341.421580] Key type ._llcrypt unregistered [ 2365.073212] Key type ._llcrypt registered [ 2365.075470] Key type .llcrypt registered [ 2365.168675] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 2366.221186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.460040] alg: No test for adler32 (adler32-zlib) [ 2367.600153] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2367.974800] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 2369.721154] Key type lgssc registered [ 2371.209781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2382.706326] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2382.769864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.816393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2403.935450] LustreError: 80780:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.142@tcp: failed processing log, type 4: rc = -110 [ 2430.636749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.683164] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2447.373794] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2455.447028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.569737] Lustre: *** cfs_fail_loc=706, val=0*** [ 2456.151079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.234056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2460.956801] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2461.169791] LustreError: 80805:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.188688] LustreError: 80805:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2465.257903] LustreError: 81865:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.274835] LustreError: 81865:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2469.762419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.110077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.515672] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2482.614825] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2488.146971] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2490.850133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2490.854633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.874776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2490.879064] Lustre: Skipped 1 previous similar message [ 2490.893457] Lustre: Skipped 1 previous similar message [ 2495.977029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2495.985662] Lustre: Skipped 1 previous similar message [ 2496.415596] Lustre: server umount lustre-OST0000 complete [ 2501.089040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.094104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.101028] Lustre: Skipped 1 previous similar message [ 2501.102976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2501.109412] Lustre: Skipped 1 previous similar message [ 2506.211046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.220407] Lustre: Skipped 1 previous similar message [ 2511.330078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.341804] Lustre: Skipped 1 previous similar message [ 2514.399170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2514.675714] Lustre: server umount lustre-MDT0000 complete [ 2516.449118] LustreError: 81872:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.463734] LustreError: 81872:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2518.438790] LustreError: 80787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849645 with bad export cookie 12146041798922513439 [ 2518.446463] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.450749] LustreError: 80787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2518.803345] Lustre: server umount lustre-MDT0001 complete [ 2529.011457] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 2531.255566] Key type lgssc unregistered [ 2531.557649] LNet: 84403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2531.564354] LNetError: 84403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2531.586390] LNet: Removed LNI 192.168.206.142@tcp [ 2532.278433] Key type .llcrypt unregistered [ 2532.279984] Key type ._llcrypt unregistered [ 2556.439686] Key type ._llcrypt registered [ 2556.442687] Key type .llcrypt registered [ 2556.530446] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 2557.469720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2557.478520] alg: No test for adler32 (adler32-zlib) [ 2558.549486] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2558.836915] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 2560.479400] Key type lgssc registered [ 2561.885516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2569.620237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2569.636874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.839931] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2582.433310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.758922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2586.533328] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2588.138208] LustreError: 86505:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.160489] LustreError: 86505:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2593.250694] LustreError: 86508:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.278906] LustreError: 86508:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2594.885755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2595.223969] LustreError: 86505:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.243846] LustreError: 86505:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2595.273404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.416571] LustreError: 86509:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.437349] LustreError: 86509:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2600.862148] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2605.539911] LustreError: 87250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.559591] LustreError: 87250:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2608.915414] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.924290] Lustre: MGS: Client 43703521-d415-42c7-ab78-2bb611140646 (at 0@lo) reconnecting [ 2608.946713] LustreError: 86519:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2609.234254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.310749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2614.167701] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2629.604739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.623450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.631968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.809058] Lustre: server umount lustre-OST0000 complete [ 2639.840643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.843614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.858754] Lustre: Skipped 2 previous similar messages [ 2639.892968] Lustre: Skipped 2 previous similar messages [ 2644.965543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.982073] Lustre: Skipped 1 previous similar message [ 2645.339422] Lustre: server umount lustre-MDT0000 complete [ 2649.806624] Lustre: server umount lustre-MDT0001 complete [ 2653.969820] Lustre: server umount MGS complete [ 2666.319538] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:56:31 (1787849791) [ 2694.408528] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 2704.469460] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 2737.226315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.684109] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2749.693195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.887670] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2749.924363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2749.975318] Lustre: lustre-MDT0001: new disk, initializing [ 2750.030042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2750.055693] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2750.068286] Lustre: 93687:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2750.084884] LustreError: 93687:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2754.016520] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2763.835217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2764.053071] Lustre: lustre-OST0000: new disk, initializing [ 2764.056409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2764.061116] Lustre: 94452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2764.137345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.654548] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2771.488349] Lustre: 94848:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003877a06a x1874696271339264/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787849914 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2778.289653] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2784.910683] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.940667] Lustre: MGS: Client 4e9ca681-cd0f-4923-97ff-72c240c5f98d (at 0@lo) reconnecting [ 2784.990231] Lustre: 92953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2785.011267] LustreError: 93673:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2785.050716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2785.113104] Lustre: lustre-MDT0000: new disk, initializing [ 2785.285631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.339071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2785.472963] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2786.406041] Lustre: 95521:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d6222374 x1874696271345792/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787849929 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2789.740954] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2791.903974] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2795.504235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2795.515783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2795.673512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2819.553399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.569267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.590091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2821.602248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.608784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.162069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.174091] Lustre: Skipped 1 previous similar message [ 2833.375318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2833.577672] Lustre: server umount lustre-OST0000 complete [ 2839.524169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.527443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2839.545211] Lustre: Skipped 1 previous similar message [ 2839.545857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2839.549781] Lustre: Skipped 3 previous similar messages [ 2843.782815] Lustre: server umount lustre-MDT0000 complete [ 2844.647760] LustreError: 93663:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.674600] LustreError: 93663:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 2853.941180] Lustre: server umount lustre-MDT0001 complete [ 2857.774202] Lustre: server umount MGS complete [ 2870.310914] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:59:55 (1787849995) [ 2889.632969] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 2892.014489] Key type lgssc unregistered [ 2892.296721] LNet: 98024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2892.302644] LNetError: 98024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2892.339225] LNet: Removed LNI 192.168.206.142@tcp [ 2893.157146] Key type .llcrypt unregistered [ 2893.164520] Key type ._llcrypt unregistered [ 2909.226494] Key type ._llcrypt registered [ 2909.229702] Key type .llcrypt registered [ 2909.313767] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 2910.152938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2910.306067] alg: No test for adler32 (adler32-zlib) [ 2911.320956] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2911.527868] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 2913.159308] Key type lgssc registered [ 2914.147041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2921.649557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2921.689532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.645957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.131509] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2934.240831] LustreError: 99248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.262120] LustreError: 99248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2936.920410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.415504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.606631] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2950.139718] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2956.192918] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2963.122086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.555230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.685464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2970.158147] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 2977.433891] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2989.536591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.546728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.565704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.609579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.619076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.635753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.574060] Lustre: server umount lustre-OST0000 complete [ 2998.759225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2998.767812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.785052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.790180] Lustre: Skipped 1 previous similar message [ 3002.848377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.865582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.942785] Lustre: server umount lustre-MDT0000 complete [ 3007.983608] LustreError: 99249:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.028395] LustreError: 99249:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3009.126374] LustreError: 99229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850135 with bad export cookie 1746810329691173152 [ 3009.131328] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3009.148307] LustreError: 99229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3009.538134] Lustre: server umount lustre-MDT0001 complete [ 3019.893049] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 3022.993442] Key type lgssc unregistered [ 3023.353306] LNet: 102799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3023.358196] LNetError: 102799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3024.426907] LNet: Removed LNI 192.168.206.142@tcp [ 3025.695281] Key type .llcrypt unregistered [ 3025.698762] Key type ._llcrypt unregistered [ 3041.914488] Key type ._llcrypt registered [ 3041.918356] Key type .llcrypt registered [ 3042.059855] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 3043.551653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3043.767764] alg: No test for adler32 (adler32-zlib) [ 3045.028684] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3045.303200] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 3047.025516] Key type lgssc registered [ 3048.176473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3055.497612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3055.502385] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3055.522126] LustreError: 103883:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3055.541668] LustreError: 103880:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3055.549034] LustreError: 103880:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3055.558464] LustreError: 103880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3055.575419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3057.432382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.682099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.941439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.339481] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3075.903430] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing lsmod [ 3078.877391] LustreError: 105087:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3078.891937] LustreError: 105088:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3078.896586] LustreError: 105087:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3078.902484] LustreError: 105087:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3078.912584] LustreError: 105087:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3078.916717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.299516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.769244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.772173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.782245] Lustre: Skipped 2 previous similar messages [ 3084.789837] Lustre: Skipped 1 previous similar message [ 3088.634175] Lustre: server umount lustre-MDT0000 complete [ 3089.890795] LustreError: 105275:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.937951] LustreError: 105275:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 3090.927052] LustreError: 103916:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.955743] LustreError: 103916:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3093.409793] LustreError: 103900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850220 with bad export cookie 3693905957135457398 [ 3093.410462] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3093.435738] LustreError: 103900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3095.136090] LustreError: 104352:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005c1a00c9 x1874696780926208/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3095.350187] Lustre: server umount lustre-MDT0001 complete [ 3110.581463] Lustre: server umount lustre-OST0000 complete [ 3119.408569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.005581] LustreError: 106154:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.137099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.424804] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3125.219027] LustreError: 106155:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.340835] LustreError: 106154:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.527110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.879022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3136.915625] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3143.576669] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3148.938711] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3155.265577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.602213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.669270] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3164.860122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3169.027211] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3178.457539] Lustre: server umount lustre-OST0000 complete [ 3184.097746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.105280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.119197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.636334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.638435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.671100] Lustre: Skipped 1 previous similar message [ 3190.753103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.761901] Lustre: Skipped 1 previous similar message [ 3194.847237] Lustre: 103285:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850305/real 1787850305] req@00000000c062ca8b x1874696780953600/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787850321 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3194.862527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.884590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.898773] Lustre: Skipped 1 previous similar message [ 3196.383117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3196.565397] Lustre: server umount lustre-MDT0000 complete [ 3196.767404] Lustre: 103284:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850307/real 1787850307] req@000000009b88abd7 x1874696780954496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787850323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3199.521189] LustreError: 106135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850326 with bad export cookie 3693905957135458084 [ 3199.527958] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3199.530461] LustreError: 106135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3199.804165] Lustre: server umount lustre-MDT0001 complete [ 3210.721666] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 3213.174911] Key type lgssc unregistered [ 3213.475748] LNet: 109689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3213.487799] LNetError: 109689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3213.525269] LNet: Removed LNI 192.168.206.142@tcp [ 3214.449195] Key type .llcrypt unregistered [ 3214.455168] Key type ._llcrypt unregistered [ 3239.675567] Key type ._llcrypt registered [ 3239.677147] Key type .llcrypt registered [ 3239.821382] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 3241.064329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3241.127377] alg: No test for adler32 (adler32-zlib) [ 3242.220193] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3242.537641] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 3244.224168] Key type lgssc registered [ 3245.631177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3253.171979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3253.188641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.591680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.363488] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3264.993411] LustreError: 111106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.014867] LustreError: 111106:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3266.003310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.404553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3270.153325] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3276.988746] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3282.206076] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3288.336135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.669340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.459676] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3299.958279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3301.663265] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3307.530817] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3310.624379] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3315.685136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3315.690831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.693083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3315.719432] Lustre: Skipped 1 previous similar message [ 3316.024637] Lustre: server umount lustre-OST0000 complete [ 3320.803774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.808051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.820462] Lustre: Skipped 1 previous similar message [ 3320.825177] Lustre: Skipped 1 previous similar message [ 3325.396766] Lustre: server umount lustre-MDT0000 complete [ 3325.920943] LustreError: 112163:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.932394] LustreError: 112163:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3328.545532] LustreError: 111088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850455 with bad export cookie 14417057134547033944 [ 3328.553923] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3328.562313] LustreError: 111088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3328.807287] Lustre: server umount lustre-MDT0001 complete [ 3339.412441] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 3341.788458] Key type lgssc unregistered [ 3342.077947] LNet: 114747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3342.086946] LNetError: 114747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3342.097654] LNet: Removed LNI 192.168.206.142@tcp [ 3342.993190] Key type .llcrypt unregistered [ 3342.996066] Key type ._llcrypt unregistered [ 3358.862745] Key type ._llcrypt registered [ 3358.866901] Key type .llcrypt registered [ 3358.971819] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 3360.060454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3360.104827] alg: No test for adler32 (adler32-zlib) [ 3361.121762] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3361.338437] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 3362.967210] Key type lgssc registered [ 3363.986197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3371.214987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3371.251689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.683386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.553683] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3383.274189] LustreError: 115964:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.293472] LustreError: 115964:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3384.773993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.087281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.303385] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3396.842309] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3402.813491] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3409.888240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3410.210890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.130972] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3417.405954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3424.993409] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3439.077403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3439.088655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.110943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.096668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3440.106940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.107802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.127566] Lustre: Skipped 1 previous similar message [ 3444.878192] Lustre: server umount lustre-OST0000 complete [ 3450.337252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.343429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3450.377234] Lustre: Skipped 1 previous similar message [ 3450.413977] Lustre: Skipped 1 previous similar message [ 3455.456578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.465504] Lustre: Skipped 1 previous similar message [ 3456.070854] Lustre: server umount lustre-MDT0000 complete [ 3458.831107] LustreError: 115944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850585 with bad export cookie 2270905907458476997 [ 3458.835165] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3458.849934] LustreError: 115944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3459.062239] Lustre: server umount lustre-MDT0001 complete [ 3469.545539] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 3472.768876] Key type lgssc unregistered [ 3473.074227] LNet: 119520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3473.095175] LNetError: 119520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3473.123118] LNet: Removed LNI 192.168.206.142@tcp [ 3473.895309] Key type .llcrypt unregistered [ 3473.897260] Key type ._llcrypt unregistered [ 3497.276359] Key type ._llcrypt registered [ 3497.278187] Key type .llcrypt registered [ 3497.461288] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 3498.570610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.603037] alg: No test for adler32 (adler32-zlib) [ 3499.719296] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3499.995474] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 3501.671208] Key type lgssc registered [ 3502.929265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3510.156674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3510.168179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.758654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.502683] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3522.023658] LustreError: 120940:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.043550] LustreError: 120940:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3525.171309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.643541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.509431] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3537.735714] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3543.592542] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3550.633942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3550.982075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.916134] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3557.119045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3565.071518] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3572.183762] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3575.756782] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3575.763145] Lustre: Skipped 2 previous similar messages [ 3580.030215] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3580.042214] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3580.055415] Lustre: Skipped 2 previous similar messages [ 3580.068649] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3585.537631] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.42@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3585.555273] Lustre: Skipped 2 previous similar messages [ 3588.888784] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3588.897309] Lustre: Skipped 1 previous similar message [ 3588.919483] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.206.42@tcp 192.168.0.[2,10]@tcp [ 3594.272953] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.42/24@tcp in log lustre-MDT0000 [ 3594.278649] Lustre: Skipped 2 previous similar messages [ 3595.900942] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.42/24@tcp [ 3595.903518] Lustre: Skipped 1 previous similar message [ 3603.429910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.438354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.450044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3603.460356] LustreError: Skipped 1 previous similar message [ 3603.475499] Lustre: Skipped 1 previous similar message [ 3607.988965] Lustre: server umount lustre-OST0000 complete [ 3612.644068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3612.656254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.678829] Lustre: Skipped 1 previous similar message [ 3612.687407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.666073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.692227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3613.697030] Lustre: Skipped 1 previous similar message [ 3618.523849] Lustre: server umount lustre-MDT0000 complete [ 3618.786028] LustreError: 121933:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3621.858383] LustreError: 120921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850748 with bad export cookie 7288616398769121905 [ 3621.861039] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.868930] LustreError: 120921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3622.312266] Lustre: server umount lustre-MDT0001 complete [ 3633.692245] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 3636.233605] Key type lgssc unregistered [ 3636.612634] LNet: 125637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3636.622355] LNetError: 125637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3637.676539] LNet: Removed LNI 192.168.206.142@tcp [ 3638.553524] Key type .llcrypt unregistered [ 3638.555573] Key type ._llcrypt unregistered [ 3668.386116] Key type ._llcrypt registered [ 3668.387936] Key type .llcrypt registered [ 3668.524768] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 3669.681336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3669.832032] alg: No test for adler32 (adler32-zlib) [ 3670.823263] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3671.101035] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 3672.799186] Key type lgssc registered [ 3673.659808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3681.060379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3681.083155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.589374] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3682.631227] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.42/24@tcp [ 3682.654137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.859366] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3693.026402] LustreError: 127199:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3693.049449] LustreError: 127199:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3694.967706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.390489] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3695.415612] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.42/24@tcp [ 3695.448755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3700.120722] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3712.996429] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3724.646366] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3733.401265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.879916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.849581] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3750.841063] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3753.333369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3768.809713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.815712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.825608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.833902] Lustre: Skipped 1 previous similar message [ 3768.841791] Lustre: Skipped 1 previous similar message [ 3773.773467] Lustre: server umount lustre-OST0000 complete [ 3777.506458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3777.511113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.523158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.043799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.044778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.075078] Lustre: Skipped 1 previous similar message [ 3783.502067] Lustre: server umount lustre-MDT0000 complete [ 3784.161066] LustreError: 127199:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3784.183900] LustreError: 127199:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3787.276404] LustreError: 127180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850913 with bad export cookie 12202167733896720787 [ 3787.286117] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3787.291393] LustreError: 127180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3787.633586] Lustre: server umount lustre-MDT0001 complete [ 3800.077104] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 3803.145716] Key type lgssc unregistered [ 3803.545770] LNet: 130811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3803.568342] LNetError: 130811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3803.600451] LNet: Removed LNI 192.168.206.142@tcp [ 3804.491977] Key type .llcrypt unregistered [ 3804.498232] Key type ._llcrypt unregistered [ 3830.258860] Key type ._llcrypt registered [ 3830.261547] Key type .llcrypt registered [ 3830.341800] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 3843.484977] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 3844.343290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3844.518240] alg: No test for adler32 (adler32-zlib) [ 3845.551383] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3845.725046] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 3847.399165] Key type lgssc registered [ 3848.365458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3878.256400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3878.278109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.489205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3879.510653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3879.571183] Lustre: lustre-MDT0000: new disk, initializing [ 3879.618846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.631552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3883.208269] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3891.803445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.906204] Lustre: 134192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3891.968956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3891.972728] Lustre: Skipped 1 previous similar message [ 3892.039819] Lustre: lustre-MDT0001: new disk, initializing [ 3892.136798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3892.172709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3892.188934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3895.940779] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3904.595748] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3909.634109] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3917.176685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.581087] Lustre: lustre-OST0000: new disk, initializing [ 3917.587677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3917.598559] Lustre: 136159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3917.691877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.231723] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3927.658559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3927.688456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3927.778925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3932.413691] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3937.233064] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3937.482695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3941.619319] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3941.814142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3948.512718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3948.518509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.524557] LustreError: Skipped 1 previous similar message [ 3948.548405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3948.556477] Lustre: Skipped 1 previous similar message [ 3953.643170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3953.649439] Lustre: Skipped 1 previous similar message [ 3958.239159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3958.375081] Lustre: server umount lustre-OST0000 complete [ 3963.873654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3963.874536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.874547] Lustre: Skipped 1 previous similar message [ 3963.876148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.000038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.014553] Lustre: Skipped 2 previous similar messages [ 3974.118448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3974.123561] Lustre: Skipped 1 previous similar message [ 3976.159312] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3976.347066] Lustre: server umount lustre-MDT0000 complete [ 3979.763442] LustreError: 134186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851106 with bad export cookie 3511011627937130172 [ 3979.772594] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3979.774222] LustreError: 134186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3980.049819] Lustre: server umount lustre-MDT0001 complete [ 3987.908224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.380964] LustreError: 138377:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.399648] LustreError: 138377:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 3988.430193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3992.574667] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 3993.576098] LustreError: 138378:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.692436] LustreError: 138377:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4000.153767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.502726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4004.924087] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4012.699314] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4017.880858] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4023.999216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4024.333042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.532197] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4035.229911] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4039.455761] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4039.637390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4043.364271] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4043.542697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4049.794621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4049.909602] Lustre: lustre-OST0001: new disk, initializing [ 4049.913282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4049.922665] Lustre: 141750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4050.000434] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4054.694739] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4055.614439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4055.625215] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4055.729816] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4062.330699] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4062.482548] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4066.092388] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4066.261503] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4070.475767] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff985f0b61c000.ost_server_uuid 50 [ 4071.723696] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff985f0b61c000.ost_server_uuid in FULL state after 0 sec [ 4081.637062] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4081.646716] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.655301] Lustre: Skipped 1 previous similar message [ 4081.658510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4081.665287] Lustre: Skipped 1 previous similar message [ 4085.034501] Lustre: server umount lustre-OST0001 complete [ 4086.770171] LustreError: 140196:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4086.801569] LustreError: 140196:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4089.826456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.836798] LustreError: Skipped 1 previous similar message [ 4089.839308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.855210] Lustre: Skipped 1 previous similar message [ 4091.884401] LustreError: 140338:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.902631] LustreError: 140338:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4095.246627] Lustre: server umount lustre-OST0000 complete [ 4102.115594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.130188] Lustre: Skipped 1 previous similar message [ 4102.143503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.150609] Lustre: Skipped 4 previous similar messages [ 4104.893437] Lustre: server umount lustre-MDT0000 complete [ 4107.234429] LustreError: 138377:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.248396] LustreError: 138377:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4108.585270] LustreError: 141749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851235 with bad export cookie 3511011627937131054 [ 4108.601373] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4108.937421] Lustre: server umount lustre-MDT0001 complete [ 4130.738759] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 4135.040378] Key type lgssc unregistered [ 4135.447594] LNet: 144755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4135.453249] LNetError: 144755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4135.468979] LNet: Removed LNI 192.168.206.142@tcp [ 4136.368198] Key type .llcrypt unregistered [ 4136.371126] Key type ._llcrypt unregistered [ 4181.880925] Key type ._llcrypt registered [ 4181.882735] Key type .llcrypt registered [ 4181.980097] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 4196.750629] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 4198.150113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4198.183351] alg: No test for adler32 (adler32-zlib) [ 4199.238164] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 4199.479638] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 4201.143567] Key type lgssc registered [ 4202.261936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4237.316287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4237.346947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.568423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4238.594545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4238.652916] Lustre: lustre-MDT0000: new disk, initializing [ 4238.731859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.746950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4242.581942] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4252.209867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.361895] Lustre: 148910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4252.421265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4252.425880] Lustre: Skipped 1 previous similar message [ 4252.497779] Lustre: lustre-MDT0001: new disk, initializing [ 4252.584313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4252.627995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4252.641209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4256.674627] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4266.177283] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.116618] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4279.323209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.729547] Lustre: lustre-OST0000: new disk, initializing [ 4279.733575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4279.751209] Lustre: 150881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4279.885634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.140801] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4288.062738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4288.079613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4288.158494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4295.687979] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4300.672471] Lustre: Failing over lustre-OST0000 [ 4300.689774] LustreError: 151687:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x2:0x0].0x0 (0000000066622805) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4302.836540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4302.841307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.912534] Lustre: server umount lustre-OST0000 complete [ 4320.227205] Lustre: 147137:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851430/real 1787851430] req@000000000b3a08bf x1874697992045056/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787851446 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4320.261365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.876582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.278965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.296359] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4325.280732] Lustre: 147137:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851435/real 1787851435] req@00000000350b5414 x1874697992046720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4328.548989] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4329.401788] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4330.463219] Lustre: 147138:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851440/real 1787851440] req@00000000ea3a3eeb x1874697992047872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4332.432629] Lustre: Failing over lustre-MDT0000 [ 4332.784239] Lustre: server umount lustre-MDT0000 complete [ 4334.496574] Lustre: 147138:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851445/real 1787851445] req@0000000007786f57 x1874697992048896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4334.561379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4334.561588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.566177] LustreError: 148920:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.566187] LustreError: 148920:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4334.660939] Lustre: Skipped 2 previous similar messages [ 4335.151488] LustreError: 148918:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.166289] LustreError: 148918:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4339.685256] LustreError: 150678:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.713094] LustreError: 150678:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4344.804629] LustreError: 148919:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.833275] LustreError: 148919:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4349.921573] LustreError: 148923:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.946270] LustreError: 148923:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4350.944142] Lustre: 147139:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851461/real 1787851461] req@00000000aa7fe648 x1874697992058496/t0(0) o400->MGC192.168.206.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787851477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4350.984135] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.104788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.707569] LustreError: 148920:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.750601] LustreError: 148920:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 4361.196212] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6e61316b6e2967d0 [ 4361.212446] Lustre: MGC192.168.206.142@tcp: Connection restored to 0@lo (at 0@lo) [ 4361.431164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4361.524396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.587774] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4365.839432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4366.307408] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4366.829321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4366.843424] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4366.912930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4377.064590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4377.073728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.075622] LustreError: Skipped 1 previous similar message [ 4377.076325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4377.091633] Lustre: Skipped 1 previous similar message [ 4377.116402] Lustre: Skipped 1 previous similar message [ 4382.185104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4382.191022] Lustre: Skipped 1 previous similar message [ 4387.298652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4387.301652] Lustre: Skipped 1 previous similar message [ 4388.320956] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4388.528514] Lustre: server umount lustre-OST0000 complete [ 4397.537448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.557226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.492020] Lustre: server umount lustre-MDT0000 complete [ 4402.658634] LustreError: 148919:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.685707] LustreError: 148919:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4402.912263] LustreError: 148903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851529 with bad export cookie 7953692754392475600 [ 4402.916866] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4402.926419] LustreError: 148903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4403.187537] Lustre: server umount lustre-MDT0001 complete [ 4417.152225] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 4420.679980] Key type lgssc unregistered [ 4421.112696] LNet: 154544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4421.122909] LNetError: 154544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4421.139768] LNet: Removed LNI 192.168.206.142@tcp [ 4422.224463] Key type .llcrypt unregistered [ 4422.226736] Key type ._llcrypt unregistered [ 4460.505523] Key type ._llcrypt registered [ 4460.507833] Key type .llcrypt registered [ 4460.612925] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 4461.343103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4461.459242] alg: No test for adler32 (adler32-zlib) [ 4462.546121] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 4462.863292] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 4464.625744] Key type lgssc registered [ 4465.822072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4474.266673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4474.288600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.050424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.742725] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4485.603580] LustreError: 156064:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.635767] LustreError: 156064:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4490.730314] LustreError: 156065:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4491.106220] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4494.601739] LustreError: 156093:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4494.619769] LustreError: 156093:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4494.634650] LustreError: 156093:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4494.967703] Lustre: server umount lustre-MDT0000 complete [ 4510.244520] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 4521.925636] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 4568.916964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.150341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.188689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.303186] Lustre: lustre-MDT0000: new disk, initializing [ 4569.458758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.508585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4575.099505] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4586.648771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.841166] Lustre: 159397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4586.918706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4586.922778] Lustre: Skipped 1 previous similar message [ 4586.924949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4587.025807] Lustre: lustre-MDT0001: new disk, initializing [ 4587.149969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4587.221076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4587.242049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4592.482393] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4608.433963] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4617.358732] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4625.803600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4626.299323] Lustre: lustre-OST0000: new disk, initializing [ 4626.307615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4626.332528] Lustre: 161369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4626.566158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.233833] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 4635.746103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4635.751688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4635.875302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4644.498347] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4649.235911] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4651.070543] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:29:36 (1787851776) [ 4927.973834] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.975162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4928.001412] Lustre: Skipped 2 previous similar messages [ 4928.014190] Lustre: Skipped 2 previous similar messages [ 4933.092087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.098052] Lustre: Skipped 2 previous similar messages [ 4933.675969] Lustre: server umount lustre-MDT0000 complete [ 4937.801462] LustreError: 159391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852064 with bad export cookie 12381717776438182292 [ 4937.803214] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4937.808732] LustreError: 159391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4938.214833] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.219785] LustreError: 159406:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.233188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4942.239870] LustreError: 161793:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.240065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4944.195543] Lustre: server umount lustre-MDT0001 complete [ 4962.271143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4962.576062] Lustre: server umount lustre-OST0000 complete [ 4972.619959] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 4987.178587] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 5037.805410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.039617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5038.073602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5038.239349] Lustre: lustre-MDT0000: new disk, initializing [ 5038.433667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.501444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5044.101446] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5058.182025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.288126] Lustre: 166074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5058.372764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5058.376926] Lustre: Skipped 1 previous similar message [ 5058.493366] Lustre: lustre-MDT0001: new disk, initializing [ 5058.620591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5058.695168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5058.716833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5063.876741] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5073.494413] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5079.710667] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5086.199387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.408334] Lustre: lustre-OST0000: new disk, initializing [ 5086.413586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5086.421524] Lustre: 168045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5086.520853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.928062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5092.940398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5093.130466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5093.138054] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5102.977483] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5107.695092] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5107.995170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5113.197919] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5113.560936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5118.948597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5118.949243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.973933] LustreError: Skipped 1 previous similar message [ 5118.979445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.023890] Lustre: Skipped 1 previous similar message [ 5122.054819] Lustre: server umount lustre-OST0000 complete [ 5129.209781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.228358] Lustre: Skipped 1 previous similar message [ 5132.853257] Lustre: server umount lustre-MDT0000 complete [ 5134.304154] LustreError: 166085:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.323614] LustreError: 166085:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5136.797083] LustreError: 166066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852263 with bad export cookie 12381717776438341122 [ 5136.801174] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.813898] LustreError: 166066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5137.087686] Lustre: server umount lustre-MDT0001 complete [ 5147.668173] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:37:52 (1787852272) [ 5165.387896] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 5175.222664] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 5212.792352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.159782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5213.216245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5213.301486] Lustre: lustre-MDT0000: new disk, initializing [ 5213.397027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5213.411621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5217.680572] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5227.852048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.025087] Lustre: 172800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5228.247799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5233.152050] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5244.466438] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5249.824430] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5257.879249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5258.292859] Lustre: lustre-OST0000: new disk, initializing [ 5258.298719] Lustre: Skipped 1 previous similar message [ 5258.308603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5258.319599] Lustre: Skipped 2 previous similar messages [ 5258.330193] Lustre: 174778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5265.979828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5266.004212] Lustre: Skipped 1 previous similar message [ 5266.018244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5266.146901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5267.357029] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5278.339268] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5286.884393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5286.889580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.902626] LustreError: Skipped 1 previous similar message [ 5286.903941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5286.903945] Lustre: Skipped 3 previous similar messages [ 5286.955495] Lustre: Skipped 1 previous similar message [ 5292.974813] Lustre: server umount lustre-OST0000 complete [ 5300.193723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5300.197374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5300.238094] Lustre: Skipped 1 previous similar message [ 5303.322482] Lustre: server umount lustre-MDT0000 complete [ 5305.314408] LustreError: 174122:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5305.326572] LustreError: 174122:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5307.696578] LustreError: 172793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852434 with bad export cookie 12381717776438342004 [ 5307.708106] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5307.708714] LustreError: 172793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5308.218782] Lustre: server umount lustre-MDT0001 complete [ 5318.556620] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:40:43 (1787852443) [ 5335.941918] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 5345.162254] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 5389.754669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.268906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5390.365339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5390.485395] Lustre: lustre-MDT0000: new disk, initializing [ 5390.574627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.580858] Lustre: Skipped 2 previous similar messages [ 5390.605108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5395.980850] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5410.166486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.386405] Lustre: 179104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5410.895046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5415.129688] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5425.272322] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5431.411959] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5440.850561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5441.347694] Lustre: 181080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5450.027399] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5451.374737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5451.569162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5461.972221] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5472.230689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.239554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.240406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5472.248244] Lustre: Skipped 1 previous similar message [ 5472.259265] Lustre: Skipped 6 previous similar messages [ 5476.009785] Lustre: server umount lustre-OST0000 complete [ 5482.469077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5482.478829] LustreError: Skipped 1 previous similar message [ 5485.881919] Lustre: server umount lustre-MDT0000 complete [ 5487.588948] LustreError: 179109:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.599188] LustreError: 179109:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5489.888691] LustreError: 182186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852616 with bad export cookie 12381717776438343089 [ 5489.891620] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.896070] LustreError: 182186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5490.102690] Lustre: server umount lustre-MDT0001 complete [ 5503.832302] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 5509.264511] Key type lgssc unregistered [ 5509.626849] LNet: 182943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5509.637941] LNetError: 182943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5509.654207] LNet: Removed LNI 192.168.206.142@tcp [ 5510.749257] Key type .llcrypt unregistered [ 5510.751787] Key type ._llcrypt unregistered [ 5538.284953] Key type ._llcrypt registered [ 5538.286909] Key type .llcrypt registered [ 5538.395679] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 5540.422177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5540.506384] alg: No test for adler32 (adler32-zlib) [ 5541.774905] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5542.292642] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 5544.119191] Key type lgssc registered [ 5545.973310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5555.575390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5555.627668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.430781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.168160] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5567.995844] LustreError: 184359:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.008434] LustreError: 184359:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5572.709754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.090966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5573.320599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5578.715837] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5588.316674] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.395456] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5602.519766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5603.012958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.038771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5609.316276] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5616.809305] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5639.141174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5639.150604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.169827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.980769] Lustre: server umount lustre-OST0000 complete [ 5649.378926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5649.381710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5649.387719] Lustre: Skipped 2 previous similar messages [ 5649.396864] Lustre: Skipped 2 previous similar messages [ 5652.744928] Lustre: server umount lustre-MDT0000 complete [ 5654.496096] LustreError: 184359:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.523957] LustreError: 184359:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5657.131572] LustreError: 184341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852783 with bad export cookie 16000928730683702835 [ 5657.133428] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5657.139041] LustreError: 184341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5657.462872] Lustre: server umount lustre-MDT0001 complete [ 5671.053966] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 5673.802226] Key type lgssc unregistered [ 5674.098166] LNet: 187923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.105840] LNetError: 187923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5674.123853] LNet: Removed LNI 192.168.206.142@tcp [ 5674.982191] Key type .llcrypt unregistered [ 5674.985044] Key type ._llcrypt unregistered [ 5701.045305] Key type ._llcrypt registered [ 5701.047899] Key type .llcrypt registered [ 5701.192462] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 5702.284557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5702.367981] alg: No test for adler32 (adler32-zlib) [ 5704.347670] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5704.846764] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 5706.639754] Key type lgssc registered [ 5708.551820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5718.265417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5718.286378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.041667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5725.292870] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5730.273327] LustreError: 189339:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.290522] LustreError: 189339:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5733.605839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.056502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5738.968162] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5746.589513] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5752.488596] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5759.964924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5760.328980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.483159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5767.846620] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5776.325239] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5788.689904] Lustre: lustre-OST0000: Not available for connect from 192.168.206.42@tcp (stopping) [ 5789.157186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5789.157439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.528322] Lustre: server umount lustre-OST0000 complete [ 5798.669045] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff985f06259800.ost_server_uuid 50 [ 5800.084664] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5813.730158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5813.745126] Lustre: Skipped 1 previous similar message [ 5813.750632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.758436] Lustre: Skipped 3 previous similar messages [ 5818.848912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5818.857354] Lustre: Skipped 1 previous similar message [ 5820.128269] Lustre: server umount lustre-MDT0000 complete [ 5823.980345] LustreError: 189333:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5824.005835] LustreError: 189333:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5824.131979] LustreError: 189320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852950 with bad export cookie 5609660410186806206 [ 5824.135771] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5824.146168] LustreError: 189320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5824.574574] Lustre: server umount lustre-MDT0001 complete [ 5832.107692] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:49:17 (1787852957) [ 5837.377572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.759605] LustreError: 193252:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.839276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.749488] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5842.915790] LustreError: 193253:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.033708] LustreError: 193252:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.240951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.696995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5854.042651] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5861.064650] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5867.469428] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5874.742633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5875.071561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.177762] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5883.251223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5887.879055] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5894.034060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5894.232486] Lustre: lustre-OST0001: new disk, initializing [ 5894.235607] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5894.246557] Lustre: 196092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5894.376859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5900.919528] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 5904.026661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5904.039900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5904.200777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5909.834560] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.080653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5914.091807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.108065] Lustre: Skipped 1 previous similar message [ 5914.117104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5919.060242] Lustre: server umount lustre-OST0000 complete [ 5919.731387] LustreError: 195549:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5919.757488] LustreError: 195549:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5928.443488] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5928.710429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5933.315426] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5933.547282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5936.648904] LustreError: 195076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.684980] LustreError: 195076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 5950.434332] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5950.446213] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.450636] LustreError: Skipped 1 previous similar message [ 5950.452712] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5950.452718] Lustre: Skipped 2 previous similar messages [ 5950.472640] Lustre: Skipped 2 previous similar messages [ 5954.475590] Lustre: server umount lustre-OST0001 complete [ 5960.680147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.683885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5960.716174] Lustre: Skipped 1 previous similar message [ 5960.723943] Lustre: Skipped 1 previous similar message [ 5964.470907] Lustre: server umount lustre-MDT0000 complete [ 5968.022282] LustreError: 193232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853094 with bad export cookie 5609660410186807277 [ 5968.032361] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.038087] LustreError: 193232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5968.280561] Lustre: server umount lustre-MDT0001 complete [ 5991.190148] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:51:56 (1787853116) [ 5997.985518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.392787] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5998.446240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5998.582444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.603424] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6012.703691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.789558] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6012.822756] Lustre: 199535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6017.946891] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6026.071685] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6031.937663] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6039.673446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6039.884896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6046.760559] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6047.920940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6053.964084] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6060.110298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6060.160628] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6066.086831] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6069.932294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6072.889060] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6078.432773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6078.445842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.469931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.473761] Lustre: Skipped 1 previous similar message [ 6081.797362] Lustre: server umount lustre-OST0000 complete [ 6083.602462] LustreError: 202297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.612522] LustreError: 202297:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 6100.448786] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6100.460135] LustreError: Skipped 1 previous similar message [ 6100.464038] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.480230] Lustre: Skipped 1 previous similar message [ 6103.375791] Lustre: server umount lustre-OST0001 complete [ 6110.176137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6110.186779] LustreError: Skipped 1 previous similar message [ 6111.202606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.217351] Lustre: Skipped 8 previous similar messages [ 6113.376475] Lustre: server umount lustre-MDT0000 complete [ 6117.274115] LustreError: 199526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853243 with bad export cookie 5609660410186808684 [ 6117.276468] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.295234] LustreError: 199526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6117.731949] Lustre: server umount lustre-MDT0001 complete [ 6140.873497] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:54:26 (1787853266) [ 6157.018779] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 6166.215859] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 6211.879346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.162561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6212.204204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6212.306603] Lustre: lustre-MDT0000: new disk, initializing [ 6212.422651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6212.432657] Lustre: Skipped 3 previous similar messages [ 6212.450150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6216.860731] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6226.858331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6226.971805] Lustre: 207630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6227.079746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6227.082940] Lustre: Skipped 1 previous similar message [ 6227.161849] Lustre: lustre-MDT0001: new disk, initializing [ 6227.345960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6227.366573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6232.258592] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6241.985978] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6247.278682] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6253.175117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6253.446824] Lustre: lustre-OST0000: new disk, initializing [ 6253.452526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6253.464964] Lustre: 209600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6259.494815] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6260.279315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6260.292838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6260.393557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6268.195861] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6272.778836] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6272.950569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6277.463630] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6277.722729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6281.192995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6281.197136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.200688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6281.203254] LustreError: Skipped 1 previous similar message [ 6281.220526] Lustre: Skipped 4 previous similar messages [ 6285.695568] Lustre: server umount lustre-OST0000 complete [ 6295.825292] Lustre: server umount lustre-MDT0000 complete [ 6296.544715] LustreError: 207637:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6296.565376] LustreError: 207637:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 13 previous similar messages [ 6299.503194] LustreError: 207622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853426 with bad export cookie 5609660410186810469 [ 6299.505573] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.516250] LustreError: 207622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6299.900522] Lustre: server umount lustre-MDT0001 complete [ 6309.486034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.492597] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6322.623976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6327.090285] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6334.341626] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6339.961670] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6346.904138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.235934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.245217] Lustre: Skipped 4 previous similar messages [ 6353.811617] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6361.952946] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6367.559277] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6367.780634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6372.372097] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6372.579220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6374.889691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6374.896370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.913856] Lustre: Skipped 3 previous similar messages [ 6380.517739] Lustre: server umount lustre-OST0000 complete [ 6386.750401] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6391.635158] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6420.930277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6427.431221] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6429.547799] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.558116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.558859] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6429.588735] Lustre: Skipped 1 previous similar message [ 6434.150229] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6439.085970] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6444.966528] Lustre: server umount lustre-OST0000 complete [ 6450.145089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6450.158872] Lustre: Skipped 8 previous similar messages [ 6459.451911] Lustre: server umount lustre-MDT0000 complete [ 6460.383425] Lustre: 188652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853571/real 1787853571] req@00000000108dba93 x1874699570371584/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787853587 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6460.388164] LustreError: 212546:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.439284] LustreError: 212546:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 6463.068508] LustreError: 211796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853589 with bad export cookie 5609660410186811351 [ 6463.068888] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.076768] LustreError: 211796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6463.414885] Lustre: server umount lustre-MDT0001 complete [ 6470.111658] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:59:55 (1787853595) [ 6475.927395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.542764] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6490.076482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6494.571349] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6500.905602] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6506.257490] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6512.644718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6519.548452] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6527.047289] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6531.939187] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6532.096496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6537.172735] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6537.399251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6544.316538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6544.614844] Lustre: lustre-OST0001: new disk, initializing [ 6544.627242] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6544.639501] Lustre: 221167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6550.599389] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6553.183783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6553.197081] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6553.290904] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6560.001597] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6565.076683] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6565.294535] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6569.617737] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6569.784590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6574.051610] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.067650] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.082575] Lustre: Skipped 3 previous similar messages [ 6577.802137] Lustre: server umount lustre-OST0001 complete [ 6583.176838] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6583.435621] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6588.113517] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6588.393746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6616.494690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6616.717631] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6616.726392] Lustre: Skipped 5 previous similar messages [ 6622.133284] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6622.184358] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6622.202350] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6622.213703] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6622.232427] Lustre: Skipped 1 previous similar message [ 6628.855152] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.264372] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6646.241212] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6646.380173] Lustre: server umount lustre-OST0001 complete [ 6668.122633] Lustre: server umount lustre-OST0000 complete [ 6678.071825] Lustre: server umount lustre-MDT0000 complete [ 6681.464243] LustreError: 217764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853808 with bad export cookie 5609660410186812310 [ 6681.472907] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6681.476026] LustreError: 217764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6688.011379] Lustre: server umount lustre-MDT0001 complete [ 6711.558699] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:03:56 (1787853836) [ 6719.364752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.943480] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6719.980344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6726.408622] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6735.600280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.730929] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6735.757154] Lustre: 226409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6740.607944] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6748.489966] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6754.397114] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6761.083305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.356325] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6768.881746] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6779.268493] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6789.819436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.912682] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6797.403104] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6806.049631] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6808.084555] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6812.913702] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6813.106813] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6817.647324] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6817.870473] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6824.158924] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff985f061a3000.ost_server_uuid 50 [ 6825.576876] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff985f061a3000.ost_server_uuid in IDLE state after 0 sec [ 6826.555783] Lustre: Permanently deactivating lustre-OST0001 [ 6826.573028] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6829.962040] Lustre: Permanently reactivating lustre-OST0001 [ 6836.706105] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.734438] LustreError: Skipped 2 previous similar messages [ 6836.741860] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.772345] Lustre: Skipped 7 previous similar messages [ 6836.780372] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6836.789029] Lustre: Skipped 17 previous similar messages [ 6840.455664] Lustre: server umount lustre-OST0001 complete [ 6842.355159] LustreError: 228254:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.375596] LustreError: 228254:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 36 previous similar messages [ 6858.207439] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6858.413212] Lustre: server umount lustre-OST0000 complete [ 6868.763896] Lustre: server umount lustre-MDT0000 complete [ 6872.637846] LustreError: 226400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853999 with bad export cookie 5609660410186813605 [ 6872.652870] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6872.657127] LustreError: 226400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6873.155423] Lustre: server umount lustre-MDT0001 complete [ 6897.749474] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:07:03 (1787854023) [ 6906.655095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.998885] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6907.030875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6907.046028] Lustre: Skipped 5 previous similar messages [ 6911.762203] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6920.896920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.042050] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6921.069527] Lustre: 233102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6926.263214] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6935.179369] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.368908] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6949.768637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.088070] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6950.132493] Lustre: Permanently deactivating lustre-OST0000 [ 6950.164728] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6957.072845] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6957.320587] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6966.909873] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6976.022110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6976.162871] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6982.979388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6985.609723] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 6995.442806] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6997.580824] Lustre: Permanently reactivating lustre-OST0000 [ 6997.597872] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6997.603675] Lustre: Skipped 2 previous similar messages [ 7004.241050] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7004.243186] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp 00000000d1ea51a0, cur 1787854130 deadline 1787854113 last 1787854083 [ 7004.251714] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7004.283864] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7004.368575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7015.976766] Lustre: server umount lustre-OST0001 complete [ 7034.337197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7034.603042] Lustre: server umount lustre-OST0000 complete [ 7053.279824] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7053.498577] Lustre: server umount lustre-MDT0000 complete [ 7057.017063] LustreError: 233094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854183 with bad export cookie 5609660410186815257 [ 7057.032499] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7057.035179] LustreError: 233094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7057.537530] Lustre: server umount lustre-MDT0001 complete [ 7070.769425] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 7073.719838] Key type lgssc unregistered [ 7074.024926] LNet: 237989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7074.035480] LNetError: 237989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7074.056942] LNet: Removed LNI 192.168.206.142@tcp [ 7074.871950] Key type .llcrypt unregistered [ 7074.874139] Key type ._llcrypt unregistered [ 7099.426412] Key type ._llcrypt registered [ 7099.429245] Key type .llcrypt registered [ 7099.532943] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 7100.508383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7100.549824] alg: No test for adler32 (adler32-zlib) [ 7101.765102] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7102.120640] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 7103.839193] Key type lgssc registered [ 7105.171459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7115.365317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7115.377446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.725457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.581708] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7127.013959] LustreError: 239455:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.028016] LustreError: 239455:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7129.817341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.936766] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7129.976995] Lustre: 239446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7129.994733] Lustre: Permanently deactivating lustre-MDT0001 [ 7130.022075] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7130.101706] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7130.108397] LustreError: 239486:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7130.129134] LustreError: 239486:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7130.157456] LustreError: 239486:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7130.441209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7136.003221] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7145.762470] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7153.209291] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7159.959258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7160.352692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.389815] LustreError: 241300:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7161.413682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7166.451988] LustreError: 241377:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7166.702565] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7171.563376] LustreError: 241298:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7174.648456] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7175.523616] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7179.760970] LustreError: 241300:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.790421] LustreError: 241300:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 7182.326108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.572223] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7188.001421] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7188.019628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7189.070598] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7196.756659] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7200.880632] Lustre: Permanently reactivating lustre-MDT0001 [ 7200.906900] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7200.916167] Lustre: Skipped 2 previous similar messages [ 7202.287580] Lustre: lustre-MDT0001: Client 57a689e4-d273-4707-bfb8-4232d0edca61 (at 192.168.206.42@tcp) reconnecting [ 7209.739504] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7211.308763] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7218.458472] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7218.837485] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7223.888386] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7224.156117] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7227.605461] Lustre: Permanently deactivating lustre-MDT0001 [ 7227.659861] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7227.668590] Lustre: Skipped 2 previous similar messages [ 7237.253873] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7246.313272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.318529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7246.333821] Lustre: Skipped 2 previous similar messages [ 7246.349061] Lustre: Skipped 3 previous similar messages [ 7251.427306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7251.440564] Lustre: Skipped 3 previous similar messages [ 7251.865383] Lustre: server umount lustre-MDT0000 complete [ 7256.015597] LustreError: 243007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854382 with bad export cookie 11382504515525820048 [ 7256.016322] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7256.032114] LustreError: 243007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7256.598822] Lustre: server umount lustre-MDT0001 complete [ 7263.342918] Lustre: server umount lustre-OST0000 complete [ 7276.238264] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:13:20 (1787854400) [ 7289.056742] Lustre: server umount lustre-OST0001 complete [ 7295.671413] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 7304.990863] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 7340.436877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.696024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7340.711177] Lustre: Skipped 2 previous similar messages [ 7340.761862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7341.008918] Lustre: lustre-MDT0000: new disk, initializing [ 7341.181933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.212352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7346.326917] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7357.416343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.555681] Lustre: 247768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7357.628952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7357.634428] Lustre: Skipped 1 previous similar message [ 7357.687708] Lustre: lustre-MDT0001: new disk, initializing [ 7357.756590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.791213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7357.808184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7361.780986] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7370.825964] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7376.390807] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7383.777211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7384.020263] Lustre: lustre-OST0000: new disk, initializing [ 7384.039355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7384.060193] Lustre: 249737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7384.154893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.492123] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7393.871853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7393.884522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7393.993245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7400.316788] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.606519] LustreError: 247776:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7411.153030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.329465] Lustre: lustre-OST0001: new disk, initializing [ 7411.333702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7411.339025] Lustre: 250853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7412.610139] LustreError: 247776:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7412.649789] LustreError: 247777:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7416.901885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7416.913013] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7416.987684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7417.267627] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7422.720116] LustreError: 247777:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7426.606110] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7428.059830] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7432.672639] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.678304] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.687394] Lustre: Skipped 1 previous similar message [ 7432.696398] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7435.949346] Lustre: server umount lustre-OST0001 complete [ 7437.809735] LustreError: 250231:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.836779] LustreError: 250231:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7453.159820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.163455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.178157] Lustre: Skipped 2 previous similar messages [ 7453.195399] Lustre: Skipped 2 previous similar messages [ 7458.326395] Lustre: server umount lustre-OST0000 complete [ 7463.395042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.398329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7463.417193] Lustre: Skipped 1 previous similar message [ 7468.517863] LustreError: 247781:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7468.541778] LustreError: 247781:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 7468.751936] Lustre: server umount lustre-MDT0000 complete [ 7472.461643] LustreError: 250852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854599 with bad export cookie 11382504515525822862 [ 7472.466560] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.486779] LustreError: 250852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7478.929275] Lustre: server umount lustre-MDT0001 complete [ 7491.177679] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 7493.842667] Key type lgssc unregistered [ 7494.131714] LNet: 252890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7494.139979] LNetError: 252890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7494.161727] LNet: Removed LNI 192.168.206.142@tcp [ 7495.224254] Key type .llcrypt unregistered [ 7495.226174] Key type ._llcrypt unregistered [ 7534.321747] Key type ._llcrypt registered [ 7534.333192] Key type .llcrypt registered [ 7534.432437] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 7535.728514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7535.781797] alg: No test for adler32 (adler32-zlib) [ 7536.856791] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7537.131167] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 7538.895159] Key type lgssc registered [ 7539.900949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7547.333631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7547.363224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.672937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7548.695540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7548.836425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.532464] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7563.083795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7563.246280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7563.314461] Lustre: 255100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7563.778341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7567.755236] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7576.234598] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7583.282827] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7590.062788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7590.348203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7590.538559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.598755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7595.994370] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7603.178705] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7656.933117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7656.944366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.948368] LustreError: Skipped 1 previous similar message [ 7656.949231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7656.981705] Lustre: Skipped 1 previous similar message [ 7661.154777] Lustre: server umount lustre-OST0000 complete [ 7665.815127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7679.178925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.511867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.950694] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7693.482984] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7696.892137] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7696.920126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7696.923356] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7696.952406] Lustre: Skipped 1 previous similar message [ 7732.710964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.714064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.721357] LustreError: Skipped 1 previous similar message [ 7732.727062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.738398] Lustre: Skipped 1 previous similar message [ 7732.749801] Lustre: Skipped 2 previous similar messages [ 7737.834198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7737.842723] Lustre: Skipped 1 previous similar message [ 7738.671227] Lustre: server umount lustre-OST0000 complete [ 7742.944477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7742.945642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.965770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.972702] Lustre: Skipped 1 previous similar message [ 7742.985396] Lustre: Skipped 1 previous similar message [ 7748.065523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7748.077121] Lustre: Skipped 1 previous similar message [ 7748.423326] Lustre: server umount lustre-MDT0000 complete [ 7751.737732] LustreError: 255091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854878 with bad export cookie 3100504010922302468 [ 7751.741458] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7751.747700] LustreError: 255091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7752.009532] Lustre: server umount lustre-MDT0001 complete [ 7761.133673] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 7763.482501] Key type lgssc unregistered [ 7763.702782] LNet: 260527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7763.712276] LNetError: 260527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7763.731919] LNet: Removed LNI 192.168.206.142@tcp [ 7764.505124] Key type .llcrypt unregistered [ 7764.509125] Key type ._llcrypt unregistered [ 7784.833385] Key type ._llcrypt registered [ 7784.836703] Key type .llcrypt registered [ 7784.926882] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 7786.501162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7786.527596] alg: No test for adler32 (adler32-zlib) [ 7787.691183] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7787.959696] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 7789.671168] Key type lgssc registered [ 7790.648493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7799.728462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7799.765197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.478052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.521660] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7812.073564] LustreError: 261924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7812.091064] LustreError: 261924:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7816.847284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7817.290917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7821.937251] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7828.702530] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7833.685213] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7840.661083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7841.085421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.785984] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7847.221469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7853.237979] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7878.113205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7878.113790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.119364] LustreError: Skipped 1 previous similar message [ 7878.133479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.027239] Lustre: server umount lustre-OST0000 complete [ 7888.352797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.362979] Lustre: Skipped 1 previous similar message [ 7888.366870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.376091] Lustre: Skipped 2 previous similar messages [ 7893.472584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.478276] Lustre: Skipped 1 previous similar message [ 7897.570327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7897.577769] Lustre: Skipped 1 previous similar message [ 7899.103748] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7899.212711] Lustre: server umount lustre-MDT0000 complete [ 7902.648103] LustreError: 261905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855029 with bad export cookie 12814164551609590407 [ 7902.658569] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7902.661391] LustreError: 261905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7902.997152] Lustre: server umount lustre-MDT0001 complete [ 7912.203807] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 7914.458529] Key type lgssc unregistered [ 7914.699856] LNet: 266232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7914.714465] LNetError: 266232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7914.741792] LNet: Removed LNI 192.168.206.142@tcp [ 7915.457493] Key type .llcrypt unregistered [ 7915.459435] Key type ._llcrypt unregistered [ 7932.381237] Key type ._llcrypt registered [ 7932.384865] Key type .llcrypt registered [ 7932.525687] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 7933.433767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7933.525499] alg: No test for adler32 (adler32-zlib) [ 7934.640922] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7934.840242] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 7936.600040] Key type lgssc registered [ 7938.270783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7947.652612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7947.668851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.027747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.801909] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7958.501945] LustreError: 267458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.511386] LustreError: 267458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7963.617427] LustreError: 267459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7965.161113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.455533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7969.564733] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 7977.138311] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7983.213212] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7990.577925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.962404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.055136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7997.459760] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8004.943423] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.893683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8017.908765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.939611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.936983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8019.944159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.981794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8022.190982] Lustre: server umount lustre-OST0000 complete [ 8027.105881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8027.114176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.136369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8027.140804] Lustre: Skipped 1 previous similar message [ 8030.180117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.183741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8030.200344] Lustre: Skipped 1 previous similar message [ 8031.927275] Lustre: server umount lustre-MDT0000 complete [ 8035.301158] LustreError: 267458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.336643] LustreError: 267458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8035.509903] LustreError: 267439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855162 with bad export cookie 9510596010445085007 [ 8035.517517] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8035.522931] LustreError: 267439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8035.904775] Lustre: server umount lustre-MDT0001 complete [ 8046.594555] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 8049.259198] Key type lgssc unregistered [ 8049.549904] LNet: 271141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8049.558039] LNetError: 271141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8049.588492] LNet: Removed LNI 192.168.206.142@tcp [ 8050.461881] Key type .llcrypt unregistered [ 8050.465577] Key type ._llcrypt unregistered [ 8075.501596] Key type ._llcrypt registered [ 8075.504600] Key type .llcrypt registered [ 8075.618932] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 8076.505713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8076.554056] alg: No test for adler32 (adler32-zlib) [ 8077.574992] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 8077.798523] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 8079.503149] Key type lgssc registered [ 8080.797112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8088.509843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8088.535685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.910551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.059297] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8100.326730] LustreError: 272559:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.344026] LustreError: 272559:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8101.800332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.240732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8106.287660] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8113.094755] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8118.336662] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8125.249818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8125.631525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8132.392035] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8134.834844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8140.607685] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8162.784608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8162.802228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8162.821434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8165.860688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.865567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8167.982292] Lustre: server umount lustre-OST0000 complete [ 8174.048799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8174.061102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.073230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8174.077547] Lustre: Skipped 1 previous similar message [ 8176.096552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.098064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8176.121479] Lustre: Skipped 1 previous similar message [ 8177.632416] Lustre: server umount lustre-MDT0000 complete [ 8181.218052] LustreError: 272558:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8181.248857] LustreError: 272558:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8181.857963] LustreError: 272538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855308 with bad export cookie 3831735966804017507 [ 8181.877879] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8182.573765] Lustre: server umount lustre-MDT0001 complete [ 8193.387656] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 8196.048526] Key type lgssc unregistered [ 8196.274075] LNet: 276862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8196.281441] LNetError: 276862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8196.297676] LNet: Removed LNI 192.168.206.142@tcp [ 8197.161254] Key type .llcrypt unregistered [ 8197.165439] Key type ._llcrypt unregistered [ 8213.210163] Key type ._llcrypt registered [ 8213.212089] Key type .llcrypt registered [ 8213.365531] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 8214.358335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8214.393921] alg: No test for adler32 (adler32-zlib) [ 8215.413750] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 8215.695363] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 8217.367793] Key type lgssc registered [ 8218.341446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8226.808837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8226.846959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8228.581644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.854480] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8239.073395] LustreError: 278097:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.103944] LustreError: 278097:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8244.195692] LustreError: 278099:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8244.276442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.650882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8249.408967] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8256.915561] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8263.041756] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8270.294572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8270.698570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8273.783405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8276.324145] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8282.759764] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8304.608854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8304.623857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8304.639270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8305.637439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8305.641534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8305.688943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.607661] Lustre: server umount lustre-OST0000 complete [ 8315.874513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.879624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8315.890669] Lustre: Skipped 1 previous similar message [ 8315.895742] Lustre: Skipped 1 previous similar message [ 8320.994852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8321.007081] Lustre: Skipped 1 previous similar message [ 8321.333705] Lustre: server umount lustre-MDT0000 complete [ 8325.606805] LustreError: 278064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855452 with bad export cookie 12371888754208706306 [ 8325.614743] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8325.622952] LustreError: 278064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8326.103153] Lustre: server umount lustre-MDT0001 complete [ 8335.689681] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 8338.313835] Key type lgssc unregistered [ 8338.559235] LNet: 281826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8338.576995] LNetError: 281826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8338.596685] LNet: Removed LNI 192.168.206.142@tcp [ 8339.431508] Key type .llcrypt unregistered [ 8339.443431] Key type ._llcrypt unregistered [ 8349.103662] Key type ._llcrypt registered [ 8349.108408] Key type .llcrypt registered [ 8349.225678] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8365.074826] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 8381.666517] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 8383.017380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8383.074823] alg: No test for adler32 (adler32-zlib) [ 8384.138697] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 8384.407616] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 8386.151200] Key type lgssc registered [ 8387.364814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8423.745580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8423.779481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8425.065690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8425.110939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8425.274316] Lustre: lustre-MDT0000: new disk, initializing [ 8425.415934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8425.458690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8430.403971] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8443.026552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8443.190994] Lustre: 285384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8443.296888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8443.300398] Lustre: Skipped 1 previous similar message [ 8443.496579] Lustre: lustre-MDT0001: new disk, initializing [ 8443.647784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8443.706079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8443.722067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8448.362749] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8458.331579] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8464.788655] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8471.556522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.843433] Lustre: lustre-OST0000: new disk, initializing [ 8471.847453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8471.853344] Lustre: 287350:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8471.954060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.270495] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8479.306245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8479.323474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8479.464950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8488.488733] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8493.053141] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8493.251616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8498.117994] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8498.309664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8500.193912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8500.194390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.200478] LustreError: Skipped 1 previous similar message [ 8500.204059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8500.214672] Lustre: Skipped 1 previous similar message [ 8505.340826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8505.356250] Lustre: Skipped 2 previous similar messages [ 8506.320696] Lustre: server umount lustre-OST0000 complete [ 8510.432625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8510.434866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8510.440687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.449900] Lustre: Skipped 1 previous similar message [ 8510.462765] Lustre: Skipped 1 previous similar message [ 8515.553244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8515.560749] Lustre: Skipped 1 previous similar message [ 8516.254805] Lustre: server umount lustre-MDT0000 complete [ 8520.566591] LustreError: 285374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855647 with bad export cookie 2059848593784356844 [ 8520.575542] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8520.589917] LustreError: 285374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8521.268663] Lustre: server umount lustre-MDT0001 complete [ 8530.710948] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:34:16 (1787855656) [ 8537.699662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.128327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.067668] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8543.206169] LustreError: 289763:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8543.228724] LustreError: 289763:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8548.340426] LustreError: 289762:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8552.345518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8552.621430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8556.964678] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8564.207402] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8569.878652] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8577.093091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8577.599021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8584.726132] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8593.497359] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8603.105549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8603.122429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8603.154167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8608.827292] Lustre: server umount lustre-OST0000 complete [ 8614.371031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8614.379330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8614.387245] LustreError: Skipped 1 previous similar message [ 8614.390987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8614.401196] Lustre: Skipped 2 previous similar messages [ 8614.425096] Lustre: Skipped 3 previous similar messages [ 8618.867537] Lustre: server umount lustre-MDT0000 complete [ 8619.490312] LustreError: 289762:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8619.512947] LustreError: 289762:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8623.249217] LustreError: 291604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855749 with bad export cookie 2059848593784357719 [ 8623.256986] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8623.263125] LustreError: 291604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8623.568531] Lustre: server umount lustre-MDT0001 complete [ 8635.162415] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 8638.203463] Key type lgssc unregistered [ 8638.448722] LNet: 293299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8638.461530] LNetError: 293299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8638.475250] LNet: Removed LNI 192.168.206.142@tcp [ 8639.286379] Key type .llcrypt unregistered [ 8639.289103] Key type ._llcrypt unregistered [ 8665.516255] Key type ._llcrypt registered [ 8665.518910] Key type .llcrypt registered [ 8665.614234] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 8680.731441] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 8681.784104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8681.815270] alg: No test for adler32 (adler32-zlib) [ 8683.049761] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 8683.400341] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 8685.071499] Key type lgssc registered [ 8686.662673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8741.430339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8741.472362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8742.768970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8742.805481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8742.904224] Lustre: lustre-MDT0000: new disk, initializing [ 8742.972856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8742.987763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8748.255396] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8758.930815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.127625] Lustre: 297131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8759.214478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8759.218583] Lustre: Skipped 1 previous similar message [ 8759.341219] Lustre: lustre-MDT0001: new disk, initializing [ 8759.495407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8759.535445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8759.540662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8764.603593] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8775.192208] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8781.194607] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8789.169133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8789.371574] Lustre: lustre-OST2710: new disk, initializing [ 8789.374139] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8789.378786] Lustre: 299098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8789.455861] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8797.248309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8797.259944] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8797.355394] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8797.407524] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8807.288680] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8814.061928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8814.250857] Lustre: lustre-OST03e8: new disk, initializing [ 8814.253921] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8814.261840] Lustre: 300163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8814.349768] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8819.830322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8819.848423] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8819.954713] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8820.419701] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8827.454241] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8833.082079] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8833.247825] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8836.879973] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8837.068569] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8840.834076] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8841.011220] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8845.580362] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8845.830020] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8850.913590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.919041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8850.942149] Lustre: Skipped 2 previous similar messages [ 8850.947908] Lustre: Skipped 1 previous similar message [ 8852.740558] Lustre: server umount lustre-MDT0000 complete [ 8857.679937] LustreError: 299097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855984 with bad export cookie 11608910549241271545 [ 8857.682351] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8857.687758] LustreError: 299097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8858.185333] Lustre: server umount lustre-MDT0001 complete [ 8872.823882] Lustre: server umount lustre-OST2710 complete [ 8887.371656] Lustre: server umount lustre-OST03e8 complete [ 8907.237697] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 8918.017359] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 8959.467049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.719976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8959.748714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8959.847413] Lustre: lustre-MDT0000: new disk, initializing [ 8959.967835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.980973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8965.457637] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8977.781432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.920158] Lustre: 305414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8978.002358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8978.005347] Lustre: Skipped 1 previous similar message [ 8978.120806] Lustre: lustre-MDT0001: new disk, initializing [ 8978.257112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8978.317558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8978.343034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8983.269378] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 8994.652454] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9001.454720] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9009.871978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9010.266192] Lustre: lustre-OST0000: new disk, initializing [ 9010.269259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9010.274868] Lustre: 307386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9010.364801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9015.915473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9015.933413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9016.118538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9018.105229] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9028.633328] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9034.524635] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9034.794255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9039.023578] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9039.299545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9041.893872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9041.896372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9041.912541] LustreError: Skipped 1 previous similar message [ 9041.931176] Lustre: Skipped 1 previous similar message [ 9041.935268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9041.935278] Lustre: Skipped 2 previous similar messages [ 9047.029819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9047.037781] Lustre: Skipped 2 previous similar messages [ 9047.449854] Lustre: server umount lustre-OST0000 complete [ 9052.145790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.167695] Lustre: Skipped 2 previous similar messages [ 9052.171687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.249287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.263569] Lustre: Skipped 2 previous similar messages [ 9057.970788] Lustre: server umount lustre-MDT0000 complete [ 9061.999066] LustreError: 306209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856188 with bad export cookie 11608910549241273225 [ 9062.001925] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.008585] LustreError: 306209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9062.428459] Lustre: server umount lustre-MDT0001 complete [ 9072.997719] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:43:18 (1787856198) [ 9075.249092] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9077.106853] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:43:22 (1787856202) [ 9081.024691] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 9110.920210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.186793] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9111.226541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9111.486900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.176747] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9125.608792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9125.745103] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9125.767324] Lustre: 311240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9130.566484] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9152.872764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.129130] Lustre: Denying initial registration attempt from nid 192.168.206.142@tcp, specified as failover [ 9153.148019] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9153.160714] LustreError: 312697:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9153.171840] LustreError: 312697:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9153.181343] LustreError: 312697:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9153.289754] Lustre: server umount lustre-OST0000 complete [ 9153.296159] LustreError: 312697:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9161.698944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9161.708252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.717917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9166.404868] Lustre: server umount lustre-MDT0000 complete [ 9169.377565] LustreError: 312378:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.399766] LustreError: 312378:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 9169.927479] LustreError: 311234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856296 with bad export cookie 11608910549241274107 [ 9169.935146] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9169.939431] LustreError: 311234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9170.333905] Lustre: server umount lustre-MDT0001 complete [ 9182.759773] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 9185.800201] Key type lgssc unregistered [ 9186.167881] LNet: 313822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9186.175771] LNetError: 313822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9186.198935] LNet: Removed LNI 192.168.206.142@tcp [ 9187.294591] Key type .llcrypt unregistered [ 9187.303184] Key type ._llcrypt unregistered [ 9198.326865] Key type ._llcrypt registered [ 9198.329904] Key type .llcrypt registered [ 9198.439338] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 9199.871671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9199.888960] alg: No test for adler32 (adler32-zlib) [ 9200.964979] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 9201.232580] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 9202.906897] Key type lgssc registered [ 9203.854770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9253.672558] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 9264.189015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9264.244490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.590400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9265.606901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9265.770264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9270.971429] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9282.129855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.371226] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9282.433757] Lustre: 316589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9282.918562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9288.622926] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9297.584259] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9303.243152] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9323.173548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.353281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9323.562408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9329.339208] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9337.494993] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9341.921444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9341.940793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9341.955340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9347.558438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9347.570714] Lustre: Skipped 3 previous similar messages [ 9352.673444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9352.683876] Lustre: Skipped 1 previous similar message [ 9354.209492] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9354.345698] Lustre: server umount lustre-OST0000 complete [ 9359.840724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9359.851435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9359.868428] Lustre: Skipped 1 previous similar message [ 9359.880646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9362.918220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.979482] Lustre: server umount lustre-MDT0000 complete [ 9369.090620] LustreError: 316579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856495 with bad export cookie 15702920999402017600 [ 9369.099550] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9369.104611] LustreError: 316579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9369.592700] Lustre: server umount lustre-MDT0001 complete [ 9384.430348] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 9389.150188] Key type lgssc unregistered [ 9389.563709] LNet: 320168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9389.573038] LNetError: 320168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9389.597645] LNet: Removed LNI 192.168.206.142@tcp [ 9390.666680] Key type .llcrypt unregistered [ 9390.668919] Key type ._llcrypt unregistered [ 9419.303377] Key type ._llcrypt registered [ 9419.305476] Key type .llcrypt registered [ 9419.389586] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 9420.392288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9420.546864] alg: No test for adler32 (adler32-zlib) [ 9421.654323] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 9421.910193] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 9423.552641] Key type lgssc registered [ 9424.608067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9432.510344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9432.536564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9433.938618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.340185] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9444.321699] LustreError: 321585:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9444.351924] LustreError: 321585:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9447.031496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9447.511160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9452.308361] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9460.032485] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9466.049875] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9475.439447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9475.862142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9481.699781] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9488.207779] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9503.719513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9503.723644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9503.734557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9503.754862] Lustre: Skipped 1 previous similar message [ 9503.774089] Lustre: Skipped 2 previous similar messages [ 9506.350902] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.42@tcp (stopping) [ 9508.833601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9508.849432] Lustre: Skipped 2 previous similar messages [ 9509.679718] Lustre: server umount lustre-MDT0000 complete [ 9511.423911] LustreError: 323886:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9513.958942] LustreError: 322313:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.274849] LustreError: 321566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856640 with bad export cookie 10221274698408613639 [ 9514.284407] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9514.289310] LustreError: 321566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9516.547748] LustreError: 321582:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9516.573122] LustreError: 321582:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9516.599573] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.42@tcp (stopping) [ 9519.074552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.082029] Lustre: Skipped 1 previous similar message [ 9520.730642] Lustre: server umount lustre-MDT0001 complete [ 9527.806463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9536.180553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9538.925164] LustreError: 323418:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9538.945955] LustreError: 323418:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9539.003259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9539.074133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9543.384921] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9549.286400] LustreError: 323418:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9549.294394] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9549.310020] LustreError: 323418:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9551.777189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.131610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.906758] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9558.512114] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9564.384376] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9566.258245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9573.075603] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9575.371141] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9583.074251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9583.084866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.105099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9583.107685] Lustre: Skipped 1 previous similar message [ 9594.342105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9594.354717] Lustre: Skipped 5 previous similar messages [ 9596.383437] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9596.561274] Lustre: server umount lustre-OST0000 complete [ 9603.559418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9603.565598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9603.582769] Lustre: Skipped 1 previous similar message [ 9614.303668] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9614.554751] Lustre: server umount lustre-MDT0000 complete [ 9614.817606] LustreError: 325200:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9614.836388] LustreError: 325200:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9618.438606] LustreError: 321566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856745 with bad export cookie 10221274698408621234 [ 9618.450286] LustreError: 321566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9618.451366] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9618.700616] Lustre: server umount lustre-MDT0001 complete [ 9628.315068] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 9630.847714] Key type lgssc unregistered [ 9631.071985] LNet: 327879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9631.082595] LNetError: 327879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9631.108404] LNet: Removed LNI 192.168.206.142@tcp [ 9631.948492] Key type .llcrypt unregistered [ 9631.952878] Key type ._llcrypt unregistered [ 9657.776340] Key type ._llcrypt registered [ 9657.779339] Key type .llcrypt registered [ 9657.895201] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 9658.842676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9659.035177] alg: No test for adler32 (adler32-zlib) [ 9660.086144] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 9660.341079] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 9662.039230] Key type lgssc registered [ 9663.248363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9670.674914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9670.694101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.105921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9675.944350] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9681.385725] LustreError: 329297:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.408384] LustreError: 329297:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9683.757908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.108037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9688.513219] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9695.675537] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9700.571490] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9704.421651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9704.431659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.448334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9706.977096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9706.979969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.025610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9709.032733] Lustre: Skipped 1 previous similar message [ 9709.481865] Lustre: server umount lustre-MDT0000 complete [ 9712.628290] LustreError: 329277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856839 with bad export cookie 9917541903537462344 [ 9712.635099] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9712.637614] LustreError: 329277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9712.949613] Lustre: server umount lustre-MDT0001 complete [ 9727.883405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.270898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9728.299537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9728.487416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9733.072924] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9741.725291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.870879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9741.926227] Lustre: 331884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9742.235413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9747.482591] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9755.017847] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9760.310710] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.523203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.773600] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9769.784520] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9769.792544] LustreError: 333769:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9769.800863] LustreError: 333769:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9769.808211] LustreError: 333769:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9769.890812] Lustre: server umount lustre-OST0000 complete [ 9769.893831] LustreError: 333769:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9780.847805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.171808] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9781.383121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.994043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9787.282294] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9793.988824] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9801.065774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9801.181237] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9801.309453] Lustre: lustre-OST0001: new disk, initializing [ 9801.313273] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9801.319510] Lustre: 335309:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9801.404180] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9807.423610] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [ 9807.993210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9808.007472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9808.141796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9814.861929] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9818.594319] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.594374] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9818.612660] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9818.617210] Lustre: Skipped 1 previous similar message [ 9823.713973] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9823.726659] Lustre: Skipped 1 previous similar message [ 9824.410835] Lustre: server umount lustre-OST0001 complete [ 9828.323484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9828.342480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.372323] Lustre: Skipped 1 previous similar message [ 9828.835943] LustreError: 334235:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9828.861214] LustreError: 334235:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9833.963520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9833.970947] LustreError: 334234:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9833.971232] Lustre: Skipped 3 previous similar messages [ 9833.991334] LustreError: 334234:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9834.254796] Lustre: server umount lustre-OST0000 complete [ 9839.074778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9839.088673] Lustre: Skipped 2 previous similar messages [ 9843.962496] Lustre: server umount lustre-MDT0000 complete [ 9844.195517] LustreError: 331897:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.212970] LustreError: 331897:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 9847.200481] LustreError: 331878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856973 with bad export cookie 9917541903537462890 [ 9847.202442] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9847.210955] LustreError: 331878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9847.436943] Lustre: server umount lustre-MDT0001 complete [ 9856.876849] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [ 9859.441507] Key type lgssc unregistered [ 9859.681595] LNet: 337272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9859.695595] LNetError: 337272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9859.709364] LNet: Removed LNI 192.168.206.142@tcp [ 9860.413702] Key type .llcrypt unregistered [ 9860.416382] Key type ._llcrypt unregistered [ 9929.926672] Key type ._llcrypt registered [ 9929.928431] Key type .llcrypt registered [ 9930.027936] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [ 9944.387210] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [ 9945.493095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9945.531094] alg: No test for adler32 (adler32-zlib) [ 9946.954271] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 9947.433258] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [ 9949.191219] Key type lgssc registered [ 9950.370392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9987.211946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9987.248386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9988.559320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9988.612472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9988.748483] Lustre: lustre-MDT0000: new disk, initializing [ 9988.848835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9988.862503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9993.649454] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10004.634672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10004.766332] Lustre: 342204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10004.819429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10004.823453] Lustre: Skipped 1 previous similar message [10004.890475] Lustre: lustre-MDT0001: new disk, initializing [10004.996231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10005.043904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10005.062129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10009.929448] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10018.945043] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10024.345329] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10031.053405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10031.280714] Lustre: lustre-OST0000: new disk, initializing [10031.284704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10031.289457] Lustre: 344174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10031.382637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10037.557976] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10039.382996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10039.392069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10039.484476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10047.108318] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10051.623181] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10051.862341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10057.110332] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10057.354740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10060.265087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.268602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10060.276157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10060.289596] Lustre: Skipped 1 previous similar message [10065.390925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10065.398445] Lustre: Skipped 2 previous similar messages [10065.585142] Lustre: server umount lustre-OST0000 complete [10070.497915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10070.499609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.526761] Lustre: Skipped 1 previous similar message [10070.538929] Lustre: Skipped 1 previous similar message [10077.590545] Lustre: server umount lustre-MDT0000 complete [10080.740265] LustreError: 342215:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.751376] LustreError: 342215:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10081.469479] LustreError: 342195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857208 with bad export cookie 4786217289691192782 [10081.470553] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10081.480987] LustreError: 342195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10082.006658] Lustre: server umount lustre-MDT0001 complete [10091.584595] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:00:16 (1787857216) [10100.394939] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:00:25 (1787857225) [10108.928045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.268490] LustreError: 346929:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10109.339696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.838582] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10114.535320] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.629891] LustreError: 346929:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.428103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.690693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10125.452305] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10132.294358] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10137.637284] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10142.755308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10143.086874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10148.453858] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10154.406621] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10161.294801] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10163.068823] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10164.557351] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10166.143186] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10167.773298] Lustre: Failing over lustre-MDT0000 [10167.804631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10167.842310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.867308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10168.028447] Lustre: server umount lustre-MDT0000 complete [10169.824849] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.837253] LustreError: 346930:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10180.072683] LustreError: 348179:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.086439] LustreError: 348179:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [10186.207322] Lustre: 340427:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857296/real 1787857296] req@00000000b2f60d5f x1874704019293952/t0(0) o400->MGC192.168.206.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787857312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10186.233907] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10186.292906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10195.439093] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x426c0f4439e0655a [10195.457894] Lustre: MGC192.168.206.142@tcp: Connection restored to 0@lo (at 0@lo) [10195.905848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10195.961299] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10197.020521] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10200.542933] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10201.075912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10201.106519] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10201.146527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10210.053381] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10211.790404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10215.531572] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10217.386964] Lustre: Failing over lustre-MDT0000 [10217.505762] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.42@tcp (stopping) [10217.665760] Lustre: server umount lustre-MDT0000 complete [10221.538546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10221.545497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.550041] LustreError: 346925:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.553316] Lustre: Skipped 4 previous similar messages [10221.574411] LustreError: 346925:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 17 previous similar messages [10236.643729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.777274] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10236.897814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [10236.909207] Lustre: Skipped 2 previous similar messages [10237.027679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10237.064395] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10237.972433] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10241.481969] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10242.032240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10242.046166] Lustre: Skipped 2 previous similar messages [10242.080777] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10242.129099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10251.650648] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10253.296396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10262.496966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.510681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10262.513523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10262.519828] Lustre: Skipped 1 previous similar message [10266.556131] Lustre: server umount lustre-MDT0000 complete [10267.626945] LustreError: 346929:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.665336] LustreError: 346929:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [10270.807331] LustreError: 349712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857397 with bad export cookie 4786217289691195491 [10270.808169] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10270.817978] LustreError: 349712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10271.197936] Lustre: server umount lustre-MDT0001 complete [10285.864226] Lustre: server umount lustre-OST0000 complete [10303.017930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.531296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10308.187168] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10317.527748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.466235] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10330.655396] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10337.051733] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10343.087157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10343.329249] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [10343.342171] Lustre: Skipped 2 previous similar messages [10348.568979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10348.986425] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10356.147770] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10363.883563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10363.884386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.901204] LustreError: Skipped 1 previous similar message [10363.922831] Lustre: Skipped 2 previous similar messages [10366.786458] Lustre: server umount lustre-OST0000 complete [10374.113347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10377.068399] Lustre: server umount lustre-MDT0000 complete [10379.240374] LustreError: 353281:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.270545] LustreError: 353281:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10380.872640] LustreError: 353262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857507 with bad export cookie 4786217289691196240 [10380.877528] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10380.885450] LustreError: 353262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10381.353719] Lustre: server umount lustre-MDT0001 complete [10393.340746] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [10396.414357] Key type lgssc unregistered [10396.822842] LNet: 356818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10396.851530] LNetError: 356818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10396.897271] LNet: Removed LNI 192.168.206.142@tcp [10397.769178] Key type .llcrypt unregistered [10397.780867] Key type ._llcrypt unregistered [10428.010896] Key type ._llcrypt registered [10428.017764] Key type .llcrypt registered [10428.202602] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [10429.426634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10429.450383] alg: No test for adler32 (adler32-zlib) [10430.500695] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [10430.691681] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [10432.351280] Key type lgssc registered [10434.083977] Lustre: Echo OBD driver; http://www.lustre.org/ [10442.650900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10442.668988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.059996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10449.327273] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10453.491519] LustreError: 358334:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.516782] LustreError: 358334:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10458.616144] LustreError: 358335:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.582233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10459.981188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10465.190203] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10474.215932] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10481.235616] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10489.014656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10489.335886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.524507] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10504.789644] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10513.914274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10526.690663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10526.704754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10526.721795] LustreError: 359062:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.919964] Lustre: server umount lustre-MDT0000 complete [10529.252423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10529.254931] LustreError: 358335:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.274404] Lustre: Skipped 1 previous similar message [10529.299837] LustreError: 358335:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10531.028293] LustreError: 360839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857657 with bad export cookie 8753455526809053065 [10531.037532] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10531.041594] LustreError: 360839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10531.326334] Lustre: server umount lustre-MDT0001 complete [10546.328888] Lustre: server umount lustre-OST0000 complete [10560.496250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.294802] LustreError: 362069:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.353272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10566.044037] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10570.721785] LustreError: 362070:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.745339] LustreError: 362070:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10575.134573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10575.545163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10580.096209] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10587.556958] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10593.784503] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10600.623650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10601.080045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10606.214658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10607.857543] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10615.922408] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10631.137677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.144638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10631.158275] Lustre: Skipped 2 previous similar messages [10631.173811] Lustre: Skipped 2 previous similar messages [10633.808663] Lustre: server umount lustre-MDT0000 complete [10636.262335] LustreError: 362070:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10636.308269] LustreError: 362070:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [10637.938986] LustreError: 362049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857764 with bad export cookie 8753455526809054171 [10637.958723] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10638.426365] Lustre: server umount lustre-MDT0001 complete [10652.892821] Lustre: server umount lustre-OST0000 complete [10668.612295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10669.149271] LustreError: 365980:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.214801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10674.147502] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10683.208629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10688.365161] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10696.537324] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10701.854277] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10708.320649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.635680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10708.641241] Lustre: Skipped 1 previous similar message [10713.785369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10714.490837] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10721.457251] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10729.442220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10729.442758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10729.462714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10729.464548] Lustre: Skipped 1 previous similar message [10731.835232] Lustre: server umount lustre-OST0000 complete [10735.074316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10735.079843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10735.088298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10735.092152] Lustre: Skipped 1 previous similar message [10739.686465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10739.693213] Lustre: Skipped 1 previous similar message [10741.282493] Lustre: server umount lustre-MDT0000 complete [10744.807655] LustreError: 365980:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.848388] LustreError: 365980:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [10744.954376] LustreError: 366715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857871 with bad export cookie 8753455526809055200 [10744.959728] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10744.967554] LustreError: 366715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10745.443811] Lustre: server umount lustre-MDT0001 complete [10757.032116] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [10760.156640] Key type lgssc unregistered [10760.536066] LNet: 369515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10760.545732] LNetError: 369515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10761.574516] LNet: Removed LNI 192.168.206.142@tcp [10762.360610] Key type .llcrypt unregistered [10762.366207] Key type ._llcrypt unregistered [10788.192122] Key type ._llcrypt registered [10788.195311] Key type .llcrypt registered [10788.295197] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [10789.191290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10789.327327] alg: No test for adler32 (adler32-zlib) [10790.490404] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [10790.915411] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [10792.623319] Key type lgssc registered [10793.666256] Lustre: Echo OBD driver; http://www.lustre.org/ [10801.960160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10801.985259] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.996380] LustreError: 370959:0:(osd_handler.c:8550:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10802.028578] LustreError: 370959:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10802.034412] LustreError: 370959:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10802.040965] LustreError: 370959:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10802.051556] LustreError: 370959:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10809.178903] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.189545] LustreError: 371356:0:(osd_handler.c:8550:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10809.213180] LustreError: 371356:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10809.222686] LustreError: 371356:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10809.229320] LustreError: 371356:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10809.235933] LustreError: 371356:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10829.661891] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [10832.632797] Key type lgssc unregistered [10832.915545] LNet: 372363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10832.928152] LNetError: 372363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10832.949264] LNet: Removed LNI 192.168.206.142@tcp [10834.011549] Key type .llcrypt unregistered [10834.017148] Key type ._llcrypt unregistered [10854.725141] Key type ._llcrypt registered [10854.728150] Key type .llcrypt registered [10854.867667] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [10871.661171] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [10872.796648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10872.947053] alg: No test for adler32 (adler32-zlib) [10873.956441] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [10874.188936] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [10875.919210] Key type lgssc registered [10877.225601] Lustre: Echo OBD driver; http://www.lustre.org/ [10916.000806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10916.013657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10917.419942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10917.534371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10917.714311] Lustre: lustre-MDT0000: new disk, initializing [10917.872225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10917.931982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10923.819909] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10935.682143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10935.787743] Lustre: 375521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10935.848923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10935.852830] Lustre: Skipped 1 previous similar message [10935.929251] Lustre: lustre-MDT0001: new disk, initializing [10936.045900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10936.090840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10936.113414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10940.916185] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10951.269482] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10957.125834] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10963.665971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10963.911733] Lustre: lustre-OST0000: new disk, initializing [10963.915706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10963.923355] Lustre: 377491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10964.010755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10969.923389] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [10971.193439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10971.200690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10971.301156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10978.696592] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10982.636995] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10982.852627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10987.176229] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10987.411434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10992.104461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.107724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10992.114310] Lustre: Skipped 1 previous similar message [10992.119355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10992.124204] LustreError: Skipped 1 previous similar message [10995.380814] Lustre: server umount lustre-OST0000 complete [11002.342542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.357259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11002.360897] Lustre: Skipped 1 previous similar message [11002.375938] Lustre: Skipped 2 previous similar messages [11005.415253] Lustre: server umount lustre-MDT0000 complete [11009.708675] LustreError: 375513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858136 with bad export cookie 2396848985957239808 [11009.712325] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11009.730023] LustreError: 375513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11010.078814] Lustre: server umount lustre-MDT0001 complete [11018.067619] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11019.605515] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:15:45 (1787858145) [11026.203497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.623534] LustreError: 379950:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11026.640048] LustreError: 379950:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11026.686790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11031.158200] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11032.032899] LustreError: 379951:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11037.164257] LustreError: 379950:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.792809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.059108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11044.390940] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11052.191987] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11057.461797] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11063.558118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11063.913615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11070.081937] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11078.083367] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11085.381361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11085.516335] Lustre: lustre-OST0001: new disk, initializing [11085.519658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11085.524107] Lustre: 382787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11085.593403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11091.296885] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11092.106333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11092.112777] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11092.232048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11101.659307] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11107.811092] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11107.813559] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11107.838105] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11107.863849] Lustre: Skipped 1 previous similar message [11109.933401] Lustre: lustre-OST0001: Not available for connect from 192.168.206.42@tcp (stopping) [11112.768086] Lustre: server umount lustre-OST0001 complete [11112.938634] LustreError: 381773:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11112.951271] LustreError: 381773:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11118.070496] LustreError: 383182:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11118.085312] LustreError: 383182:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11128.293070] LustreError: 381772:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.316859] LustreError: 381772:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11131.872494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11131.884167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11131.893387] Lustre: Skipped 1 previous similar message [11131.900168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11136.261914] Lustre: server umount lustre-OST0000 complete [11142.627401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11142.633808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11142.646897] Lustre: Skipped 1 previous similar message [11142.651795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11142.656179] Lustre: Skipped 2 previous similar messages [11146.871563] Lustre: server umount lustre-MDT0000 complete [11148.771227] LustreError: 379946:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11148.792042] LustreError: 379946:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [11150.548470] LustreError: 379931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858277 with bad export cookie 2396848985957240690 [11150.559121] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11150.564031] LustreError: 379931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11157.050124] Lustre: server umount lustre-MDT0001 complete [11168.129675] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [11170.793949] Key type lgssc unregistered [11171.086883] LNet: 384779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11171.096569] LNetError: 384779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11171.106988] LNet: Removed LNI 192.168.206.142@tcp [11172.084518] Key type .llcrypt unregistered [11172.086056] Key type ._llcrypt unregistered [11219.133447] Key type ._llcrypt registered [11219.136841] Key type .llcrypt registered [11219.263600] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [11220.404244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11220.461088] alg: No test for adler32 (adler32-zlib) [11221.522176] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [11221.764583] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [11223.424248] Key type lgssc registered [11224.463516] Lustre: Echo OBD driver; http://www.lustre.org/ [11232.094737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11232.114996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11233.432726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11233.471609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11233.608607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11238.267916] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11246.773143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.876711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11246.900881] Lustre: 387580:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11247.181335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11251.393768] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11259.192174] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11264.952817] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11267.552427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11267.559202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11267.583641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11270.625299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11270.627393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11275.744649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11275.751160] Lustre: Skipped 2 previous similar messages [11280.866586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11280.879588] Lustre: Skipped 1 previous similar message [11281.375083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11281.653816] Lustre: server umount lustre-MDT0000 complete [11285.613711] LustreError: 387570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858412 with bad export cookie 17510978985476610716 [11285.618094] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11285.621603] LustreError: 387570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11286.000633] Lustre: server umount lustre-MDT0001 complete [11295.332743] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:20:20 (1787858420) [11302.190665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11302.862986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11307.226880] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11308.009300] LustreError: 390095:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.020991] LustreError: 390095:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11313.120325] LustreError: 390094:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11317.428384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11317.964850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11324.986633] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11336.619919] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11343.717181] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11351.681897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11351.842503] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11352.034638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11359.603155] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11368.663675] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11376.596968] Lustre: Permanently deactivating lustre-OST0000 [11376.619990] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11385.415934] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11386.757737] LustreError: 393083:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11386.770159] LustreError: 393083:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11386.773912] LustreError: 393083:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11389.873506] LustreError: 393132:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11389.890161] LustreError: 393132:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11398.460221] Lustre: server umount lustre-OST0000 complete [11401.085928] LustreError: 393381:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11401.097764] LustreError: 393381:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11404.257897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11404.281694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11404.293062] Lustre: Skipped 1 previous similar message [11412.544987] Lustre: server umount lustre-MDT0000 complete [11414.497081] LustreError: 390089:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11414.516876] LustreError: 390089:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11415.744478] LustreError: 390075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858542 with bad export cookie 17510978985476611304 [11415.746147] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11415.755629] LustreError: 390075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11426.508702] Lustre: server umount lustre-MDT0001 complete [11435.611286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.668448] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11443.423558] LustreError: 394540:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11449.234949] Lustre: 394880:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.142@tcp [11451.280450] Lustre: server umount MGS complete [11464.736917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.280136] LustreError: 395605:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11465.299407] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11465.305319] Lustre: Skipped 1 previous similar message [11465.370068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11470.167045] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11470.820208] LustreError: 395606:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.997621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.262842] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11484.453783] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11492.480212] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11497.851853] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11499.733491] Lustre: Permanently reactivating lustre-OST0000 [11499.748266] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11499.759150] Lustre: Skipped 2 previous similar messages [11528.672581] LustreError: 397432:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11535.720192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11536.059431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11536.067461] Lustre: Skipped 1 previous similar message [11541.942292] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11549.885640] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11557.627553] Lustre: server umount lustre-OST0000 complete [11562.470695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11562.489396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11562.511649] Lustre: Skipped 1 previous similar message [11562.517600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11562.525929] Lustre: Skipped 2 previous similar messages [11571.283956] Lustre: server umount lustre-MDT0000 complete [11574.919744] LustreError: 395586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858701 with bad export cookie 17510978985476612354 [11574.934300] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11574.934954] LustreError: 395586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11584.675805] Lustre: server umount lustre-MDT0001 complete [11596.258854] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [11599.129229] Key type lgssc unregistered [11599.451704] LNet: 400070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11599.469683] LNetError: 400070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11599.501194] LNet: Removed LNI 192.168.206.142@tcp [11600.431213] Key type .llcrypt unregistered [11600.433622] Key type ._llcrypt unregistered [11620.527435] Key type ._llcrypt registered [11620.529601] Key type .llcrypt registered [11620.619635] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [11635.881911] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [11636.797686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11636.820553] alg: No test for adler32 (adler32-zlib) [11637.908887] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [11638.124919] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [11639.783752] Key type lgssc registered [11640.772610] Lustre: Echo OBD driver; http://www.lustre.org/ [11679.623822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11679.647300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11680.839143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11680.884310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11680.946240] Lustre: lustre-MDT0000: new disk, initializing [11681.046405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11681.075515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11686.086658] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11697.923805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11698.069331] Lustre: 403228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11698.116933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11698.120225] Lustre: Skipped 1 previous similar message [11698.171603] Lustre: lustre-MDT0001: new disk, initializing [11698.286604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11698.334381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11698.344422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11704.034052] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11714.482292] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11721.129788] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11728.457707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11728.714249] Lustre: lustre-OST0000: new disk, initializing [11728.717497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11728.722670] Lustre: 405194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11728.792949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11734.449565] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11738.200352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11738.215044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11738.363784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11744.093885] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11748.928472] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11749.259759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11753.884073] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11754.108487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11759.080254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11759.087362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11759.087899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11759.089977] Lustre: Skipped 1 previous similar message [11759.110230] Lustre: Skipped 1 previous similar message [11764.204176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11764.215382] Lustre: Skipped 1 previous similar message [11769.318110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11770.335195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11770.525453] Lustre: server umount lustre-OST0000 complete [11774.432959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11774.449883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11774.459863] Lustre: Skipped 2 previous similar messages [11779.554802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11779.561654] Lustre: Skipped 1 previous similar message [11780.779431] Lustre: server umount lustre-MDT0000 complete [11784.275808] LustreError: 403218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858910 with bad export cookie 10764421984497047625 [11784.277822] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11784.294755] LustreError: 403218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11784.530980] Lustre: server umount lustre-MDT0001 complete [11794.256358] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11796.567227] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:28:41 (1787858921) [11803.770070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.268454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11808.657946] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11809.765465] LustreError: 407657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.787615] LustreError: 407657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11813.857123] LustreError: 407656:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11817.120474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11817.383940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11822.543561] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11830.915403] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11836.240418] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11842.450736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11842.742556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11848.982488] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11856.077180] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11858.076095] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11860.583990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11872.753231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11872.753849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11872.772424] Lustre: Skipped 1 previous similar message [11872.782804] LustreError: Skipped 1 previous similar message [11872.793582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11872.816450] Lustre: Skipped 1 previous similar message [11876.795588] Lustre: server umount lustre-OST0000 complete [11882.986330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11883.023982] Lustre: Skipped 1 previous similar message [11893.217636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11893.225202] Lustre: Skipped 5 previous similar messages [11895.263132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11895.420160] Lustre: server umount lustre-MDT0000 complete [11898.338133] LustreError: 407657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11898.355884] LustreError: 407657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11898.787421] LustreError: 407638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859025 with bad export cookie 10764421984497048507 [11898.797286] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.811479] LustreError: 407638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11899.313611] Lustre: server umount lustre-MDT0001 complete [11911.166925] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [11913.675738] Key type lgssc unregistered [11913.989213] LNet: 411388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11914.004107] LNetError: 411388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11914.031726] LNet: Removed LNI 192.168.206.142@tcp [11914.868944] Key type .llcrypt unregistered [11914.872218] Key type ._llcrypt unregistered [11940.258838] Key type ._llcrypt registered [11940.273771] Key type .llcrypt registered [11940.424704] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [11940.708362] Key type .llcrypt unregistered [11940.715227] Key type ._llcrypt unregistered [11958.906570] Key type ._llcrypt registered [11958.908140] Key type .llcrypt registered [11958.997729] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [11960.072216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11960.206684] alg: No test for adler32 (adler32-zlib) [11961.234354] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [11961.475184] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [11963.127159] Key type lgssc registered [11964.059086] Lustre: Echo OBD driver; http://www.lustre.org/ [11970.638798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11970.681764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.112566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11976.004913] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11982.305639] LustreError: 413607:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.329654] LustreError: 413607:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11984.101802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11984.359518] LustreError: 413606:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.382761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11984.384127] LustreError: 413606:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [11989.479305] LustreError: 413607:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.568842] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [11994.598393] LustreError: 413602:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11994.616787] LustreError: 413602:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [11996.664237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11997.608031] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12001.771132] LustreError: 413606:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.793302] LustreError: 413606:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [12004.317454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.681069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12009.023755] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12017.123525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12017.135327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12017.164255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12019.184691] Lustre: server umount lustre-OST0000 complete [12024.809162] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12024.819096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.838274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12025.534313] Lustre: server umount lustre-MDT0000 complete [12029.488509] LustreError: 413588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859156 with bad export cookie 4901423487358927803 [12029.488737] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12029.499738] LustreError: 413588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12039.795916] Lustre: server umount lustre-MDT0001 complete [12051.048291] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [12053.655409] Key type lgssc unregistered [12053.938173] LNet: 416756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12053.947051] LNetError: 416756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12053.962234] LNet: Removed LNI 192.168.206.142@tcp [12054.980071] Key type .llcrypt unregistered [12054.985196] Key type ._llcrypt unregistered [12082.419176] Key type ._llcrypt registered [12082.423862] Key type .llcrypt registered [12082.549345] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [12083.556207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12083.583785] alg: No test for adler32 (adler32-zlib) [12084.799299] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12084.962325] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [12086.791381] Key type lgssc registered [12088.047197] Lustre: Echo OBD driver; http://www.lustre.org/ [12097.573038] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12097.621326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12113.375444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12118.496315] LustreError: 418148:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.142@tcp: failed processing log, type 4: rc = -110 [12145.248210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12151.711156] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12160.426794] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12168.577302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12169.405315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12169.501473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12174.726157] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12174.818318] LustreError: 418173:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12174.841789] LustreError: 418173:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12179.944497] LustreError: 419188:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12179.963212] LustreError: 419188:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12183.255806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12183.585047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12187.976876] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12196.288490] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12203.000624] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12214.249784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.257171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12214.276724] Lustre: Skipped 1 previous similar message [12214.288750] Lustre: Skipped 1 previous similar message [12217.630918] Lustre: server umount lustre-OST0000 complete [12224.481813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12224.482784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12224.500391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12227.850729] Lustre: server umount lustre-MDT0000 complete [12229.611404] LustreError: 419187:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12229.632270] LustreError: 419187:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12231.837822] LustreError: 420681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859358 with bad export cookie 14380318125364308467 [12231.850523] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12231.854572] LustreError: 420681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12232.121265] Lustre: server umount lustre-MDT0001 complete [12244.029757] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [12246.721196] Key type lgssc unregistered [12246.993468] LNet: 421724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12247.000624] LNetError: 421724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12247.025779] LNet: Removed LNI 192.168.206.142@tcp [12247.844246] Key type .llcrypt unregistered [12247.846430] Key type ._llcrypt unregistered [12275.936113] Key type ._llcrypt registered [12275.938058] Key type .llcrypt registered [12276.086889] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [12277.557483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12277.601485] alg: No test for adler32 (adler32-zlib) [12278.661708] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12278.870850] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [12280.583160] Key type lgssc registered [12281.821466] Lustre: Echo OBD driver; http://www.lustre.org/ [12290.212637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12290.231529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12291.610794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12297.131594] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12301.797571] LustreError: 423140:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.838747] LustreError: 423140:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12306.924320] LustreError: 423141:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.237572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12308.760576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12314.729274] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12323.844583] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.186377] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12337.679941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12337.983779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12344.620459] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12352.958622] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12361.526851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12366.818101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12366.821095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12366.831736] Lustre: Skipped 1 previous similar message [12368.929331] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.42@tcp (stopping) [12368.934349] Lustre: Skipped 2 previous similar messages [12371.941326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12371.950445] Lustre: Skipped 2 previous similar messages [12372.416932] Lustre: server umount lustre-MDT0000 complete [12374.029090] LustreError: 424286:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.704175] LustreError: 423122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.42@tcp arrived at 1787859501 with bad export cookie 7463168191316531504 [12374.722438] LustreError: 423122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12376.601626] LustreError: 423120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859503 with bad export cookie 7463168191316530846 [12376.615682] LustreError: 423120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12376.616849] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12377.056874] LustreError: 423870:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12377.062709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.075159] LustreError: 423870:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12377.106574] Lustre: Skipped 1 previous similar message [12377.115740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12381.088485] LustreError: 423140:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.104113] LustreError: 423140:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12382.975973] Lustre: server umount lustre-OST0000 complete [12390.391845] Lustre: server umount lustre-MDT0001 complete [12403.728331] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [12407.158678] Key type lgssc unregistered [12407.468722] LNet: 426856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12407.476545] LNetError: 426856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12407.503640] LNet: Removed LNI 192.168.206.142@tcp [12408.237749] Key type .llcrypt unregistered [12408.241089] Key type ._llcrypt unregistered [12438.650886] Key type ._llcrypt registered [12438.654819] Key type .llcrypt registered [12438.812838] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [12439.949466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12439.973513] alg: No test for adler32 (adler32-zlib) [12441.284867] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12441.631304] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [12443.303160] Key type lgssc registered [12444.463232] Lustre: Echo OBD driver; http://www.lustre.org/ [12453.560465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12453.592947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12455.400259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12461.306573] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12464.610177] LustreError: 428273:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12464.636262] LustreError: 428273:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12469.728894] LustreError: 428274:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12472.595440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12473.162398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12479.683210] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12489.060540] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12496.198192] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12504.028416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12504.462405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12510.800533] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12518.524770] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12525.038438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12529.634158] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12529.641907] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12529.661850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12530.156396] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12530.184244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12530.188453] Lustre: Skipped 1 previous similar message [12534.040467] Lustre: server umount lustre-MDT0001 complete [12535.269482] LustreError: 428274:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.301107] LustreError: 428274:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12540.387218] LustreError: 428273:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12540.390876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12540.399278] LustreError: 428273:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12540.420495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12540.427719] Lustre: Skipped 1 previous similar message [12545.505461] LustreError: 429001:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.512630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12546.181317] Lustre: server umount lustre-OST0000 complete [12550.741322] Lustre: server umount lustre-MDT0000 complete [12565.101451] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [12567.878571] Key type lgssc unregistered [12568.180914] LNet: 431964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12568.185198] LNetError: 431964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12568.201146] LNet: Removed LNI 192.168.206.142@tcp [12569.008512] Key type .llcrypt unregistered [12569.014648] Key type ._llcrypt unregistered [12593.712694] Key type ._llcrypt registered [12593.714985] Key type .llcrypt registered [12593.801320] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [12608.508467] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [12609.230511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12609.316860] alg: No test for adler32 (adler32-zlib) [12610.405368] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12610.710782] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [12612.399894] Key type lgssc registered [12613.265421] Lustre: Echo OBD driver; http://www.lustre.org/ [12652.937359] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [12669.330074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12669.373828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12670.709302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12670.746865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12670.888323] Lustre: lustre-MDT0000: new disk, initializing [12671.018313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12671.054707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12676.751379] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12692.914415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12693.200059] Lustre: lustre-OST0000: new disk, initializing [12693.203228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12693.206847] Lustre: Skipped 1 previous similar message [12693.215144] Lustre: 436997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12693.275583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12698.753995] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12700.718081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12700.726740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12700.750702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12707.911462] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12722.552686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12722.640295] Lustre: 435977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12722.702781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12722.767495] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.42@tcp (not set up) [12722.799178] Lustre: lustre-MDT0001: new disk, initializing [12722.958131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12723.023109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12723.044920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12727.182598] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12736.231490] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12736.412554] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12740.933659] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12741.152666] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12747.235648] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12747.237399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12747.256456] Lustre: Skipped 1 previous similar message [12748.337136] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.42@tcp (stopping) [12748.344758] Lustre: Skipped 1 previous similar message [12752.154900] Lustre: server umount lustre-MDT0001 complete [12757.475290] LustreError: 435984:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12757.507970] LustreError: 435984:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [12762.593522] LustreError: 435989:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12762.622247] LustreError: 435989:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12767.725587] LustreError: 435986:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12767.745273] LustreError: 435986:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [12772.324464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12772.336678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12772.353197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12772.842839] LustreError: 435990:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12776.871739] Lustre: server umount lustre-OST0000 complete [12777.954621] LustreError: 435989:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12787.094842] Lustre: server umount lustre-MDT0000 complete [12801.218230] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [12803.676334] Key type lgssc unregistered [12803.943273] LNet: 440747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12803.947995] LNetError: 440747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12803.968682] LNet: Removed LNI 192.168.206.142@tcp [12804.855823] Key type .llcrypt unregistered [12804.859630] Key type ._llcrypt unregistered [12846.030926] Key type ._llcrypt registered [12846.033962] Key type .llcrypt registered [12846.121954] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [12846.433538] Key type .llcrypt unregistered [12846.435457] Key type ._llcrypt unregistered [12864.902170] Key type ._llcrypt registered [12864.906948] Key type .llcrypt registered [12865.082758] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [12866.451545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12866.508859] alg: No test for adler32 (adler32-zlib) [12867.683830] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12868.026273] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [12869.719208] Key type lgssc registered [12870.891132] Lustre: Echo OBD driver; http://www.lustre.org/ [12898.818825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12898.835925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12900.129465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12900.171044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12900.273694] Lustre: lustre-MDT0000: new disk, initializing [12900.369725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12900.384069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12904.991393] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12914.670684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12914.788712] Lustre: 444131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12914.840641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12914.845806] Lustre: Skipped 1 previous similar message [12914.919238] Lustre: lustre-MDT0001: new disk, initializing [12915.013926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12915.052431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12915.061598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12918.876100] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12927.739643] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12932.990894] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12939.572247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12939.780791] Lustre: lustre-OST0000: new disk, initializing [12939.784768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12939.789799] Lustre: 446098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12939.859498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12946.546899] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [12950.098532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12950.128255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12950.258502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12956.667169] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12965.867716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12965.878151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12965.887702] Lustre: Skipped 2 previous similar messages [12965.892527] Lustre: Skipped 1 previous similar message [12970.079806] Lustre: server umount lustre-MDT0000 complete [12974.550182] LustreError: 444124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860101 with bad export cookie 3691406786203698345 [12974.562796] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.563994] LustreError: 444124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12975.276868] Lustre: server umount lustre-MDT0001 complete [12989.963757] Lustre: server umount lustre-OST0000 complete [12999.932129] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:48:45 (1787860125) [13007.414884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13007.762802] LustreError: 448127:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13007.779744] LustreError: 448127:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [13007.833350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13012.168652] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [13012.963847] LustreError: 448128:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13018.081190] LustreError: 448127:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13019.620170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13019.953651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13024.074786] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [13030.900383] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13036.572768] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13056.119462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13056.447920] Lustre: Found index 0 for lustre-OST0000, updating log [13056.684220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13064.167603] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [13071.878446] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13079.219206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13087.713769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13087.716303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.741542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13087.750035] Lustre: Skipped 1 previous similar message [13088.816579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13088.826387] Lustre: Skipped 2 previous similar messages [13094.883970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13094.887439] Lustre: Skipped 1 previous similar message [13101.023095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13101.030725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13101.058339] Lustre: Skipped 1 previous similar message [13101.328855] Lustre: server umount lustre-OST0000 complete [13117.415389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13117.423882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13117.430084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13117.475302] Lustre: Skipped 2 previous similar messages [13119.967437] LNet: 1 local NIs in recovery (showing 1): 192.168.206.142@tcp [13123.629713] Lustre: server umount lustre-MDT0000 complete [13127.182344] LustreError: 448108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860253 with bad export cookie 3691406786203699738 [13127.193845] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13127.201060] LustreError: 448108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13127.556934] Lustre: server umount lustre-MDT0001 complete [13135.967248] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 15:51:01 (1787860261) [13755.757150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13756.313765] LustreError: 454569:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13756.388223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13760.785438] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [13761.510146] LustreError: 454570:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13765.603753] LustreError: 454569:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.611733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13770.040391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13774.770099] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [13783.044118] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13789.253139] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13809.121899] LNet: 1 local NIs in recovery (showing 1): 192.168.206.142@tcp [13809.834491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13810.071021] Lustre: Found index 0 for lustre-OST0000, updating log [13810.436049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13817.498694] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [13826.339525] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13829.492417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13829.504743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13829.510445] Lustre: Skipped 3 previous similar messages [13834.714429] Lustre: server umount lustre-OST0000 complete [13839.844781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13839.862229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13844.003086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13844.008764] Lustre: Skipped 8 previous similar messages [13852.200260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13852.209486] Lustre: Skipped 15 previous similar messages [13853.151283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13853.300404] Lustre: server umount lustre-MDT0000 complete [13854.248587] LustreError: 455813:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13854.272517] LustreError: 455813:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13856.502463] LustreError: 454549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860983 with bad export cookie 3691406786203700823 [13856.507268] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13856.514388] LustreError: 454549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13856.860802] Lustre: server umount lustre-MDT0001 complete [13868.512826] LNet: 1 local NIs in recovery (showing 1): 192.168.206.142@tcp [13885.945454] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [13894.928442] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [13933.482049] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:04:18 (1787861058) [13952.385793] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [13955.080963] Key type lgssc unregistered [13955.352625] LNet: 461181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13955.363195] LNetError: 461181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14071.082854] LNet: Removed LNI 192.168.206.142@tcp [14071.919680] Key type .llcrypt unregistered [14071.921627] Key type ._llcrypt unregistered [14089.264487] Key type ._llcrypt registered [14089.266288] Key type .llcrypt registered [14089.379239] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [14090.339099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14090.402288] alg: No test for adler32 (adler32-zlib) [14091.820069] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [14092.121187] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [14093.799185] Key type lgssc registered [14095.418615] Lustre: Echo OBD driver; http://www.lustre.org/ [14100.370661] LNet: Added LNI 192.168.206.142@tcp100 [8/256/0/180] [14101.169514] LNet: Added LNI 192.168.206.142@tcp101 [8/256/0/180] [14101.996684] LNet: Added LNI 192.168.206.142@tcp102 [8/256/0/180] [14103.022111] LNet: Added LNI 192.168.206.142@tcp103 [8/256/0/180] [14103.982910] LNet: Added LNI 192.168.206.142@tcp104 [8/256/0/180] [14104.928647] LNet: Added LNI 192.168.206.142@tcp105 [8/256/0/180] [14105.949311] LNet: Added LNI 192.168.206.142@tcp106 [8/256/0/180] [14106.959209] LNet: Added LNI 192.168.206.142@tcp107 [8/256/0/180] [14107.810767] LNet: Added LNI 192.168.206.142@tcp108 [8/256/0/180] [14108.926039] LNet: Added LNI 192.168.206.142@tcp109 [8/256/0/180] [14110.124531] LNet: Added LNI 192.168.206.142@tcp110 [8/256/0/180] [14111.387895] LNet: Added LNI 192.168.206.142@tcp111 [8/256/0/180] [14112.417348] LNet: Added LNI 192.168.206.142@tcp112 [8/256/0/180] [14113.428034] LNet: Added LNI 192.168.206.142@tcp113 [8/256/0/180] [14114.439676] LNet: Added LNI 192.168.206.142@tcp114 [8/256/0/180] [14115.566623] LNet: Added LNI 192.168.206.142@tcp115 [8/256/0/180] [14116.453017] LNet: Added LNI 192.168.206.142@tcp116 [8/256/0/180] [14117.376682] LNet: Added LNI 192.168.206.142@tcp117 [8/256/0/180] [14118.281742] LNet: Added LNI 192.168.206.142@tcp118 [8/256/0/180] [14119.224381] LNet: Added LNI 192.168.206.142@tcp119 [8/256/0/180] [14120.106712] LNet: Added LNI 192.168.206.142@tcp120 [8/256/0/180] [14120.936576] LNet: Added LNI 192.168.206.142@tcp121 [8/256/0/180] [14121.859419] LNet: Added LNI 192.168.206.142@tcp122 [8/256/0/180] [14122.763372] LNet: Added LNI 192.168.206.142@tcp123 [8/256/0/180] [14123.577912] LNet: Added LNI 192.168.206.142@tcp124 [8/256/0/180] [14124.456255] LNet: Added LNI 192.168.206.142@tcp125 [8/256/0/180] [14125.345092] LNet: Added LNI 192.168.206.142@tcp126 [8/256/0/180] [14126.235671] LNet: Added LNI 192.168.206.142@tcp127 [8/256/0/180] [14127.220565] LNet: Added LNI 192.168.206.142@tcp128 [8/256/0/180] [14128.075859] LNet: Added LNI 192.168.206.142@tcp129 [8/256/0/180] [14129.048040] LNet: Added LNI 192.168.206.142@tcp130 [8/256/0/180] [14130.158996] LNet: Added LNI 192.168.206.142@tcp131 [8/256/0/180] [14130.957703] LNet: Added LNI 192.168.206.142@tcp132 [8/256/0/180] [14131.924963] LNet: Added LNI 192.168.206.142@tcp133 [8/256/0/180] [14132.908882] LNet: Added LNI 192.168.206.142@tcp134 [8/256/0/180] [14133.937363] LNet: Added LNI 192.168.206.142@tcp135 [8/256/0/180] [14140.417099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14140.473516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14141.613539] Lustre: 464234:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.142@tcp: can't add NID 192.168.206.142@tcp130: rc = -75 [14141.688486] LNetError: 464261:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14141.725787] LNetError: Protocol error connecting to 192.168.206.142@tcp121 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14141.753328] LNetError: Skipped 6 previous similar messages [14141.831525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14141.871931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14141.987820] Lustre: lustre-MDT0000: new disk, initializing [14142.082398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14142.099804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14142.305473] LNet: 1 local NIs in recovery (showing 1): 192.168.206.142@tcp121 [14142.308344] LNetError: 464256:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14142.320588] LNetError: 464256:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14142.326244] LNetError: Protocol error connecting to 192.168.206.142@tcp107 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14144.353511] LNetError: 464256:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14144.366312] LNetError: 464256:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14144.377832] LNetError: Protocol error connecting to 192.168.206.142@tcp121 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14144.394380] LNetError: Skipped 4 previous similar messages [14146.178648] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14148.448488] LNetError: 461694:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14148.465660] LNetError: Protocol error connecting to 192.168.206.142@tcp121 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14156.494405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14156.622299] Lustre: 464250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14156.641429] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14156.648842] LNetError: Protocol error connecting to 192.168.206.142@tcp121 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14156.676810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14156.679897] Lustre: Skipped 1 previous similar message [14156.755957] Lustre: lustre-MDT0001: new disk, initializing [14156.795405] Lustre: 465053:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.142@tcp: can't add NID 192.168.206.142@tcp130: rc = -75 [14156.803210] Lustre: 465053:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14156.938032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14156.978515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14156.993503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14162.076202] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14163.994594] Lustre: 465398:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.142@tcp: can't add NID 192.168.206.142@tcp130: rc = -75 [14164.008885] Lustre: 465398:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14172.016813] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14173.025323] LNetError: 464260:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14173.045893] LNetError: Protocol error connecting to 192.168.206.142@tcp126 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14177.425663] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14185.136350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14185.503125] Lustre: lustre-OST0000: new disk, initializing [14185.506226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14185.512731] Lustre: 466226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14185.641501] Lustre: 466231:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.142@tcp: can't add NID 192.168.206.142@tcp130: rc = -75 [14185.649540] Lustre: 466231:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14185.712216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14190.435855] LNetError: 464262:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14190.455754] LNetError: 464262:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14190.464698] LNetError: Protocol error connecting to 192.168.206.142@tcp127 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14190.473678] LNetError: Skipped 11 previous similar messages [14191.183425] Lustre: 466512:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.142@tcp: can't add NID 192.168.206.142@tcp130: rc = -75 [14191.197426] Lustre: 466512:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14191.243314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14191.256293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14191.417444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14192.765994] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14202.908982] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14212.066960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14212.070567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14212.075997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14212.087370] Lustre: Skipped 1 previous similar message [14217.184994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14217.202798] Lustre: Skipped 2 previous similar messages [14222.306614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14222.321609] Lustre: Skipped 1 previous similar message [14225.376847] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14225.688070] Lustre: server umount lustre-OST0000 complete [14230.370392] LNetError: 464262:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.142 [14230.386458] LNetError: 464262:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [14230.392521] LNetError: Protocol error connecting to 192.168.206.142@tcp117 at host 192.168.206.142:988: is it running a compatible version of Lustre? [14230.403839] LNetError: Skipped 20 previous similar messages [14232.545642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14232.561065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14235.932592] Lustre: server umount lustre-MDT0000 complete [14239.834169] LustreError: 464243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861366 with bad export cookie 12235875599433740650 [14239.841064] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14239.845624] LustreError: 464243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14240.146261] Lustre: server umount lustre-MDT0001 complete [14252.623947] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [14255.606512] Key type lgssc unregistered [14256.089232] LNet: 468146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14256.114515] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14256.124129] LNet: Removed LNI 192.168.206.142@tcp [14257.183178] LNet: Removed LNI 192.168.206.142@tcp100 [14257.189731] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14257.197679] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14258.207288] LNet: Removed LNI 192.168.206.142@tcp101 [14258.211919] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14258.226551] LNet: Removed LNI 192.168.206.142@tcp102 [14259.295734] LNet: Removed LNI 192.168.206.142@tcp103 [14260.319197] LNet: Removed LNI 192.168.206.142@tcp104 [14260.333335] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14260.347235] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14261.407218] LNet: Removed LNI 192.168.206.142@tcp105 [14262.433026] LNet: Removed LNI 192.168.206.142@tcp106 [14263.456292] LNet: Removed LNI 192.168.206.142@tcp107 [14264.479498] LNet: Removed LNI 192.168.206.142@tcp108 [14264.483509] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14264.492541] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [14264.509305] LNet: Removed LNI 192.168.206.142@tcp109 [14265.568182] LNet: Removed LNI 192.168.206.142@tcp110 [14266.591180] LNet: Removed LNI 192.168.206.142@tcp111 [14267.615166] LNet: Removed LNI 192.168.206.142@tcp112 [14268.639164] LNet: Removed LNI 192.168.206.142@tcp113 [14269.669124] LNet: Removed LNI 192.168.206.142@tcp114 [14270.690411] LNet: Removed LNI 192.168.206.142@tcp115 [14271.711214] LNet: Removed LNI 192.168.206.142@tcp116 [14272.735238] LNet: Removed LNI 192.168.206.142@tcp117 [14272.745106] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14272.756776] LNetError: 468146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14272.771922] LNet: Removed LNI 192.168.206.142@tcp118 [14273.826209] LNet: Removed LNI 192.168.206.142@tcp119 [14274.911163] LNet: Removed LNI 192.168.206.142@tcp120 [14275.935304] LNet: Removed LNI 192.168.206.142@tcp121 [14276.962324] LNet: Removed LNI 192.168.206.142@tcp122 [14277.984995] LNet: Removed LNI 192.168.206.142@tcp123 [14279.007783] LNet: Removed LNI 192.168.206.142@tcp124 [14280.031163] LNet: Removed LNI 192.168.206.142@tcp125 [14281.055261] LNet: Removed LNI 192.168.206.142@tcp126 [14282.079161] LNet: Removed LNI 192.168.206.142@tcp127 [14283.103174] LNet: Removed LNI 192.168.206.142@tcp128 [14284.127264] LNet: Removed LNI 192.168.206.142@tcp129 [14284.131323] LNet: Removed LNI 192.168.206.142@tcp130 [14284.140639] LNet: Removed LNI 192.168.206.142@tcp131 [14284.150824] LNet: Removed LNI 192.168.206.142@tcp132 [14284.161809] LNet: Removed LNI 192.168.206.142@tcp133 [14284.169755] LNet: Removed LNI 192.168.206.142@tcp134 [14284.179114] LNet: Removed LNI 192.168.206.142@tcp135 [14284.911923] Key type .llcrypt unregistered [14284.915688] Key type ._llcrypt unregistered [14308.594814] Key type ._llcrypt registered [14308.596877] Key type .llcrypt registered [14308.774191] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [14324.684460] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [14326.322362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14326.375777] alg: No test for adler32 (adler32-zlib) [14327.615221] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [14327.864845] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [14329.528298] Key type lgssc registered [14330.925870] Lustre: Echo OBD driver; http://www.lustre.org/ [14370.263646] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:11:35 (1787861495) [14388.526605] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [14391.103937] Key type lgssc unregistered [14391.431749] LNet: 472238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14391.447041] LNetError: 472238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14391.473491] LNet: Removed LNI 192.168.206.142@tcp [14392.394424] Key type .llcrypt unregistered [14392.396894] Key type ._llcrypt unregistered [14407.778220] Key type ._llcrypt registered [14407.784254] Key type .llcrypt registered [14407.926100] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [14409.260991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14409.286908] alg: No test for adler32 (adler32-zlib) [14410.312261] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [14410.541993] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [14412.167817] Key type lgssc registered [14413.264159] Lustre: Echo OBD driver; http://www.lustre.org/ [14422.931304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14422.984861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14424.368814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14424.411812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14424.544599] Lustre: lustre-MDT0000: new disk, initializing [14424.709241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14424.735215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14429.715713] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14440.647068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14440.800559] Lustre: 473452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14440.856361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14440.860649] Lustre: Skipped 1 previous similar message [14440.943495] Lustre: lustre-MDT0001: new disk, initializing [14441.049555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14441.117488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14441.133215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14445.908226] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14456.046169] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14462.013467] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14468.631805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14468.978587] Lustre: lustre-OST0000: new disk, initializing [14468.987985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14469.000361] Lustre: 475421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14469.123494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14475.449426] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14476.862092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14476.871868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14477.018398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14484.514414] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14488.662532] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing lnet_if_list [14493.549504] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14493.581272] LNet: Added LNI 192.168.206.142@tcp10 [8/256/0/180] [14498.470492] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14498.493662] LNet: Added LNI 192.168.206.142@tcp11 [8/256/0/180] [14502.578864] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14502.608627] LNet: Added LNI 192.168.206.142@tcp12 [8/256/0/180] [14506.821414] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14506.868623] LNet: Added LNI 192.168.206.142@tcp13 [8/256/0/180] [14511.125135] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14511.151725] LNet: Added LNI 192.168.206.142@tcp14 [8/256/0/180] [14515.889415] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14515.912069] LNet: Added LNI 192.168.206.142@tcp15 [8/256/0/180] [14521.456206] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14521.503786] LNet: Added LNI 192.168.206.142@tcp16 [8/256/0/180] [14529.471808] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14529.521486] LNet: Added LNI 192.168.206.142@tcp17 [8/256/0/180] [14534.132282] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14534.161800] LNet: Added LNI 192.168.206.142@tcp18 [8/256/0/180] [14539.042542] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14539.064746] LNet: Added LNI 192.168.206.142@tcp19 [8/256/0/180] [14544.121694] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14544.152446] LNet: Added LNI 192.168.206.142@tcp20 [8/256/0/180] [14550.519650] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14550.539944] LNet: Added LNI 192.168.206.142@tcp21 [8/256/0/180] [14555.614322] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14555.703042] LNet: Added LNI 192.168.206.142@tcp22 [8/256/0/180] [14561.491683] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14561.518954] LNet: Added LNI 192.168.206.142@tcp23 [8/256/0/180] [14566.547772] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14566.588233] LNet: Added LNI 192.168.206.142@tcp24 [8/256/0/180] [14571.866406] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14571.909753] LNet: Added LNI 192.168.206.142@tcp25 [8/256/0/180] [14578.021362] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14578.063139] LNet: Added LNI 192.168.206.142@tcp26 [8/256/0/180] [14583.840539] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14583.869366] LNet: Added LNI 192.168.206.142@tcp27 [8/256/0/180] [14589.557309] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14589.598863] LNet: Added LNI 192.168.206.142@tcp28 [8/256/0/180] [14595.420619] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14595.445168] LNet: Added LNI 192.168.206.142@tcp29 [8/256/0/180] [14600.640245] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14600.676316] LNet: Added LNI 192.168.206.142@tcp30 [8/256/0/180] [14606.074457] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14606.103571] LNet: Added LNI 192.168.206.142@tcp31 [8/256/0/180] [14611.752494] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14611.784530] LNet: Added LNI 192.168.206.142@tcp32 [8/256/0/180] [14617.764445] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14617.787697] LNet: Added LNI 192.168.206.142@tcp33 [8/256/0/180] [14623.913161] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14623.931856] LNet: Added LNI 192.168.206.142@tcp34 [8/256/0/180] [14630.073519] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14630.121454] LNet: Added LNI 192.168.206.142@tcp35 [8/256/0/180] [14636.035328] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14636.059584] LNet: Added LNI 192.168.206.142@tcp36 [8/256/0/180] [14642.773093] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14642.802107] LNet: Added LNI 192.168.206.142@tcp37 [8/256/0/180] [14650.750807] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14650.779450] LNet: Added LNI 192.168.206.142@tcp38 [8/256/0/180] [14658.844214] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14658.877603] LNet: Added LNI 192.168.206.142@tcp39 [8/256/0/180] [14665.530857] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14665.554461] LNet: Added LNI 192.168.206.142@tcp40 [8/256/0/180] [14672.531049] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14672.553782] LNet: Added LNI 192.168.206.142@tcp41 [8/256/0/180] [14680.505980] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14680.564751] LNet: Added LNI 192.168.206.142@tcp42 [8/256/0/180] [14688.335369] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14688.377304] LNet: Added LNI 192.168.206.142@tcp43 [8/256/0/180] [14695.396282] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14695.433515] LNet: Added LNI 192.168.206.142@tcp44 [8/256/0/180] [14702.857407] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14702.882289] LNet: Added LNI 192.168.206.142@tcp45 [8/256/0/180] [14710.477786] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14710.510488] LNet: Added LNI 192.168.206.142@tcp46 [8/256/0/180] [14718.087316] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14718.119405] LNet: Added LNI 192.168.206.142@tcp47 [8/256/0/180] [14725.847533] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14725.872293] LNet: Added LNI 192.168.206.142@tcp48 [8/256/0/180] [14734.944689] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14734.972620] LNet: Added LNI 192.168.206.142@tcp49 [8/256/0/180] [14743.682207] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14743.715092] LNet: Added LNI 192.168.206.142@tcp50 [8/256/0/180] [14763.505987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14763.523104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14763.544318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14767.974651] Lustre: server umount lustre-MDT0000 complete [14771.996331] LustreError: 479202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861898 with bad export cookie 17752653099959212508 [14771.997873] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14772.004491] LustreError: 479202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14772.379989] Lustre: server umount lustre-MDT0001 complete [14789.603220] Lustre: 472725:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861900/real 1787861900] req@00000000b9a15b0f x1874708699185024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14789.641280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14789.653296] Lustre: Skipped 2 previous similar messages [14790.068600] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [14791.711207] Lustre: 472725:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861902/real 1787861902] req@00000000fa50570b x1874708699185152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14794.719434] Lustre: 472724:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861905/real 1787861905] req@0000000089de4d95 x1874708699185280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14797.859089] Lustre: 472724:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861908/real 1787861908] req@00000000a33e1539 x1874708699185536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14798.734165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14803.935173] Lustre: 472726:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787861914/real 1787861914] req@0000000073a24b0a x1874708699185664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787861930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14817.320536] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf65e17efd3ce644e [14817.340180] Lustre: MGC192.168.206.142@tcp: Connection restored to 0@lo (at 0@lo) [14817.611591] LustreError: 475415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14817.636445] LustreError: 475415:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [14817.692651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14818.722069] LustreError: 475416:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14818.739831] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14820.768325] LustreError: 475415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14821.861314] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14825.960044] LustreError: 475416:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14830.092594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14830.468359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14834.620321] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14835.639421] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [14843.822892] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14850.344267] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14871.523284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14871.543252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14871.565392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14871.573118] Lustre: Skipped 3 previous similar messages [14874.082422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14874.085749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14874.092061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14880.752929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14880.763859] Lustre: Skipped 3 previous similar messages [14885.343920] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14885.792779] Lustre: server umount lustre-OST0000 complete [14892.398418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14892.457597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14897.120295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14899.015469] Lustre: server umount lustre-MDT0000 complete [14902.755700] LustreError: 499670:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14902.802906] LustreError: 499670:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [14904.805878] LustreError: 479202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862031 with bad export cookie 17752653099959215182 [14904.809594] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14904.829653] LustreError: 479202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14905.248217] Lustre: server umount lustre-MDT0001 complete [14918.353618] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [14921.707255] Key type lgssc unregistered [14922.150697] LNet: 502471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14922.172693] LNetError: 502471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14922.178691] LNet: Removed LNI 192.168.206.142@tcp [14922.181640] LNet: Removed LNI 192.168.206.142@tcp10 [14922.183745] LNet: Removed LNI 192.168.206.142@tcp11 [14922.188333] LNet: Removed LNI 192.168.206.142@tcp12 [14922.194323] LNet: Removed LNI 192.168.206.142@tcp13 [14922.198630] LNet: Removed LNI 192.168.206.142@tcp14 [14922.201898] LNet: Removed LNI 192.168.206.142@tcp15 [14922.228329] LNet: Removed LNI 192.168.206.142@tcp16 [14922.230490] LNet: Removed LNI 192.168.206.142@tcp17 [14922.239747] LNet: Removed LNI 192.168.206.142@tcp18 [14922.243734] LNet: Removed LNI 192.168.206.142@tcp19 [14922.246366] LNet: Removed LNI 192.168.206.142@tcp20 [14922.248667] LNet: Removed LNI 192.168.206.142@tcp21 [14922.250759] LNet: Removed LNI 192.168.206.142@tcp22 [14922.253629] LNet: Removed LNI 192.168.206.142@tcp23 [14922.255950] LNet: Removed LNI 192.168.206.142@tcp24 [14922.258941] LNet: Removed LNI 192.168.206.142@tcp25 [14922.261215] LNet: Removed LNI 192.168.206.142@tcp26 [14922.263900] LNet: Removed LNI 192.168.206.142@tcp27 [14922.266487] LNet: Removed LNI 192.168.206.142@tcp28 [14922.268967] LNet: Removed LNI 192.168.206.142@tcp29 [14922.271837] LNet: Removed LNI 192.168.206.142@tcp30 [14922.274695] LNet: Removed LNI 192.168.206.142@tcp31 [14922.279670] LNet: Removed LNI 192.168.206.142@tcp32 [14922.282962] LNet: Removed LNI 192.168.206.142@tcp33 [14922.287209] LNet: Removed LNI 192.168.206.142@tcp34 [14922.294074] LNet: Removed LNI 192.168.206.142@tcp35 [14922.305639] LNet: Removed LNI 192.168.206.142@tcp36 [14922.314897] LNet: Removed LNI 192.168.206.142@tcp37 [14922.321903] LNet: Removed LNI 192.168.206.142@tcp38 [14922.325406] LNet: Removed LNI 192.168.206.142@tcp39 [14922.331511] LNet: Removed LNI 192.168.206.142@tcp40 [14922.337947] LNet: Removed LNI 192.168.206.142@tcp41 [14922.342597] LNet: Removed LNI 192.168.206.142@tcp42 [14922.346727] LNet: Removed LNI 192.168.206.142@tcp43 [14922.352291] LNet: Removed LNI 192.168.206.142@tcp44 [14922.356668] LNet: Removed LNI 192.168.206.142@tcp45 [14922.364533] LNet: Removed LNI 192.168.206.142@tcp46 [14922.371853] LNet: Removed LNI 192.168.206.142@tcp47 [14922.374981] LNet: Removed LNI 192.168.206.142@tcp48 [14922.378752] LNet: Removed LNI 192.168.206.142@tcp49 [14922.405424] LNet: Removed LNI 192.168.206.142@tcp50 [14923.441845] Key type .llcrypt unregistered [14923.443643] Key type ._llcrypt unregistered [14947.088583] Key type ._llcrypt registered [14947.091921] Key type .llcrypt registered [14947.225703] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [14964.208861] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [14965.571232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14965.606522] alg: No test for adler32 (adler32-zlib) [14966.594235] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [14966.828278] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [14968.567334] Key type lgssc registered [14969.878472] Lustre: Echo OBD driver; http://www.lustre.org/ [15006.101247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15006.114806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15007.366668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15007.428781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15007.533573] Lustre: lustre-MDT0000: new disk, initializing [15007.635977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15007.663711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15012.293237] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15023.097915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15023.280319] Lustre: 505671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15023.333404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15023.340599] Lustre: Skipped 1 previous similar message [15023.462149] Lustre: lustre-MDT0001: new disk, initializing [15023.597831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15023.702963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15023.723478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15027.909865] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15038.080551] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15044.998590] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15054.271439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15054.434338] Lustre: *** cfs_fail_loc=90f, val=0*** [15054.495595] Lustre: lustre-OST0000: new disk, initializing [15054.499316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15054.503486] Lustre: 507689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15054.602342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15061.255884] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15061.609785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15061.631618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15061.759059] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15071.482803] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15082.468809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15082.470620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15082.485947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15082.490463] LustreError: Skipped 1 previous similar message [15082.499300] Lustre: Skipped 1 previous similar message [15087.598437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15087.609756] Lustre: Skipped 1 previous similar message [15092.716409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15092.722333] Lustre: Skipped 1 previous similar message [15094.239470] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15094.549496] Lustre: server umount lustre-OST0000 complete [15100.386031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15100.397582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15100.411868] Lustre: Skipped 1 previous similar message [15100.439041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15102.950588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15104.676704] Lustre: server umount lustre-MDT0000 complete [15109.095203] LustreError: 505663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862235 with bad export cookie 7307779759576280206 [15109.096896] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15109.106100] LustreError: 505663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15109.584817] Lustre: server umount lustre-MDT0001 complete [15119.964754] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [15123.094945] Key type lgssc unregistered [15123.453795] LNet: 509601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15123.470492] LNetError: 509601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15123.501125] LNet: Removed LNI 192.168.206.142@tcp [15124.531275] Key type .llcrypt unregistered [15124.537586] Key type ._llcrypt unregistered [15145.417509] Key type ._llcrypt registered [15145.421340] Key type .llcrypt registered [15145.505758] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [15146.858073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15146.878073] alg: No test for adler32 (adler32-zlib) [15148.029518] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [15148.330884] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [15150.031463] Key type lgssc registered [15151.172691] Lustre: Echo OBD driver; http://www.lustre.org/ [15158.859455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15158.886190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15160.489502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15165.505429] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15171.043520] LustreError: 510872:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15171.091258] LustreError: 510872:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [15175.805306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15176.160368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15176.358816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15182.229141] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15191.838931] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15199.917679] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15208.414099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15208.767778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15217.108230] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15224.782770] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15246.284889] loop0: detected capacity change from 0 to 204800000 [15246.504249] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15246.511440] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15246.518528] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15252.932971] loop0: detected capacity change from 0 to 204800000 [15253.121736] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15253.141632] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15253.167238] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15258.321412] loop0: detected capacity change from 0 to 204800000 [15264.065240] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15264.203711] Lustre: fs15246-OST0000: new disk, initializing [15264.207634] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15264.213208] Lustre: 514489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15264.283677] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15270.498359] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15279.806064] loop1: detected capacity change from 0 to 204800000 [15285.540719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15285.675698] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15285.749301] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15285.822742] Lustre: fs15246-MDT0000: new disk, initializing [15285.958784] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15285.984056] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15286.095362] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15287.137981] Lustre: 515612:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c6ed25a4 x1874709471873536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787862429 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15291.072382] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15292.193608] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15292.210808] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15292.241263] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15316.453272] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15316.484299] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15321.049670] Lustre: server umount fs15246-MDT0000 complete [15321.591818] LustreError: 510869:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15326.703805] LustreError: 510868:0:(ldlm_lib.c:1200:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15330.844355] Lustre: server umount fs15246-OST0000 complete [15344.096361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15344.106823] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15344.127236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15345.120556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15345.130872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15347.182427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15347.187305] Lustre: Skipped 2 previous similar messages [15348.892499] Lustre: server umount lustre-OST0000 complete [15355.874507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15355.886323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15355.909099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15358.720617] Lustre: server umount lustre-MDT0000 complete [15362.529492] LustreError: 510868:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15362.553351] LustreError: 510868:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [15362.631756] LustreError: 510854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862489 with bad export cookie 9061641317380525307 [15362.641312] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15362.957583] Lustre: server umount lustre-MDT0001 complete [15372.673449] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [15375.169668] Key type lgssc unregistered [15375.485304] LNet: 518713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15375.493881] LNetError: 518713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15375.518315] LNet: Removed LNI 192.168.206.142@tcp [15376.664925] Key type .llcrypt unregistered [15376.667347] Key type ._llcrypt unregistered [15395.152402] Key type ._llcrypt registered [15395.154765] Key type .llcrypt registered [15395.370598] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [15396.763589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15396.790994] alg: No test for adler32 (adler32-zlib) [15398.140244] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [15398.562098] LNet: Added LNI 192.168.206.142@tcp [8/256/0/180] [15400.240147] Key type lgssc registered [15401.589540] Lustre: Echo OBD driver; http://www.lustre.org/ [15446.021789] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [15456.765691] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [15495.148726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15495.204683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15496.500054] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15496.554316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15496.690511] Lustre: lustre-MDT0000: new disk, initializing [15496.781659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15496.797170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15501.286613] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15512.476606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15512.640205] Lustre: 523182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15512.763109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15512.772906] Lustre: Skipped 1 previous similar message [15512.930695] Lustre: lustre-MDT0001: new disk, initializing [15513.041138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15513.069591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15513.074985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15518.522667] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15529.032829] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15535.767799] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15543.149135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15543.355197] Lustre: lustre-OST0000: new disk, initializing [15543.358403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15543.363240] Lustre: 525150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15543.457687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15550.129968] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_default_debug -1 all [15553.094058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15553.103176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15553.216469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15560.414492] Lustre: DEBUG MARKER: oleg642-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15565.934552] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15566.173343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15570.775342] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15570.980995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15573.985267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15573.990100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15574.002029] Lustre: Skipped 1 previous similar message [15574.017453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15579.127936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15579.139806] Lustre: Skipped 2 previous similar messages [15584.230478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15584.239844] Lustre: Skipped 1 previous similar message [15587.295859] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15587.499965] Lustre: server umount lustre-OST0000 complete [15594.467687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15594.487880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15594.496445] Lustre: Skipped 1 previous similar message [15597.397158] Lustre: server umount lustre-MDT0000 complete [15601.590873] LustreError: 523174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862728 with bad export cookie 3125812178475239256 [15601.612760] LustreError: 523174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15601.615587] LustreError: MGC192.168.206.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15601.963315] Lustre: server umount lustre-MDT0001 complete [15630.442937] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing set_hostid [15643.248510] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing load_modules_local [15679.127513] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15421 sec ========= 16:33:24 (1787862804) [15680.948939] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:33:26 (1787862806) === [15684.094857] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:33:29 (1787862809) === [15726.884956] Lustre: DEBUG MARKER: oleg642-server.virtnet: executing unload_modules_local [15730.091056] Key type lgssc unregistered [15730.400486] LNet: 531802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15730.422024] LNetError: 531802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15730.436769] LNet: Removed LNI 192.168.206.142@tcp [15731.419243] Key type .llcrypt unregistered [15731.421405] Key type ._llcrypt unregistered