[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500898782 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008467] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009052] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.012163] LSM: Security Framework initializing [ 0.013062] Yama: becoming mindful. [ 0.014052] SELinux: Initializing. [ 0.015093] *** VALIDATE selinux *** [ 0.024050] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030152] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031146] *** VALIDATE tmpfs *** [ 0.032506] *** VALIDATE proc *** [ 0.033285] *** VALIDATE cgroup *** [ 0.034015] *** VALIDATE cgroup2 *** [ 0.035293] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038034] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.043229] debug: unmapping init [mem 0xffffffff94e59000-0xffffffff94e60fff] [ 0.045257] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046906] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048020] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.055275] rcu: Hierarchical SRCU implementation. [ 0.059379] smp: Bringing up secondary CPUs ... [ 0.060832] x86: Booting SMP configuration: [ 0.061038] .... node #0, CPUs: #1 #2 #3 [ 0.069301] smp: Brought up 1 node, 4 CPUs [ 0.071045] smpboot: Max logical packages: 1 [ 0.072023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.177900] node 0 deferred pages initialised in 102ms [ 0.181362] devtmpfs: initialized [ 0.182257] x86/mm: Memory block size: 128MB [ 0.184819] gcov: version magic: 0x41383552 [ 0.187412] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.193165] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.196515] pinctrl core: initialized pinctrl subsystem [ 0.199280] [ 0.199854] ************************************************************* [ 0.202015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.204075] ** ** [ 0.207012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.210012] ** ** [ 0.212008] ** This means that this kernel is built to expose internal ** [ 0.213008] ** IOMMU data structures, which may compromise security on ** [ 0.216011] ** your system. ** [ 0.218006] ** ** [ 0.219006] ** If you see this message and you are not debugging the ** [ 0.222015] ** kernel, report this immediately to your vendor! ** [ 0.223018] ** ** [ 0.226012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228011] ************************************************************* [ 0.232079] NET: Registered protocol family 16 [ 0.234549] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.237065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244106] cpuidle: using governor menu [ 0.246918] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.250459] PCI: Using configuration type 1 for base access [ 0.253117] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.263103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.265075] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268086] cryptd: max_cpu_qlen set to 1000 [ 0.271061] ACPI: Added _OSI(Module Device) [ 0.272008] ACPI: Added _OSI(Processor Device) [ 0.274010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.276012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282948] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.290556] ACPI: Interpreter enabled [ 0.292057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.295012] ACPI: Using IOAPIC for interrupt routing [ 0.297180] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.301601] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.313035] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.316083] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.325084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.331681] acpiphp: Slot [2] registered [ 0.334113] acpiphp: Slot [5] registered [ 0.335142] acpiphp: Slot [6] registered [ 0.337098] acpiphp: Slot [7] registered [ 0.339082] acpiphp: Slot [8] registered [ 0.341091] acpiphp: Slot [9] registered [ 0.343086] acpiphp: Slot [10] registered [ 0.344073] acpiphp: Slot [3] registered [ 0.346103] acpiphp: Slot [4] registered [ 0.348116] acpiphp: Slot [11] registered [ 0.350068] acpiphp: Slot [12] registered [ 0.352088] acpiphp: Slot [13] registered [ 0.354105] acpiphp: Slot [14] registered [ 0.356082] acpiphp: Slot [15] registered [ 0.357083] acpiphp: Slot [16] registered [ 0.358069] acpiphp: Slot [17] registered [ 0.360064] acpiphp: Slot [18] registered [ 0.363137] acpiphp: Slot [19] registered [ 0.365151] acpiphp: Slot [20] registered [ 0.367083] acpiphp: Slot [21] registered [ 0.369109] acpiphp: Slot [22] registered [ 0.370070] acpiphp: Slot [23] registered [ 0.371062] acpiphp: Slot [24] registered [ 0.373074] acpiphp: Slot [25] registered [ 0.374079] acpiphp: Slot [26] registered [ 0.375079] acpiphp: Slot [27] registered [ 0.376066] acpiphp: Slot [28] registered [ 0.378070] acpiphp: Slot [29] registered [ 0.379143] acpiphp: Slot [30] registered [ 0.381081] acpiphp: Slot [31] registered [ 0.382077] PCI host bridge to bus 0000:00 [ 0.383011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.386014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.388012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.392014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.395019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.396159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.400000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.403399] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.414867] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.421820] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.426016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.428019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.430009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.432568] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434807] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.437030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.438815] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.456013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.461015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.469333] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.477070] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.485015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.500012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.517000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.527011] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.534021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.550029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.562626] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.589065] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.613018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.625849] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.642016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.657013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.697014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716994] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.725019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.734019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.750021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.762000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.771018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.778018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.797018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.804000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.808563] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.810321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.813373] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.815269] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.820006] iommu: Default domain type: Passthrough [ 0.820473] SCSI subsystem initialized [ 0.822107] ACPI: bus type USB registered [ 0.824122] usbcore: registered new interface driver usbfs [ 0.825061] usbcore: registered new interface driver hub [ 0.827074] usbcore: registered new device driver usb [ 0.829171] pps_core: LinuxPPS API ver. 1 registered [ 0.831008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.834064] PTP clock support registered [ 0.836139] EDAC MC: Ver: 3.0.0 [ 0.837077] PCI: Using ACPI for IRQ routing [ 0.839019] NetLabel: Initializing [ 0.840008] NetLabel: domain hash size = 128 [ 0.842009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.844086] NetLabel: unlabeled traffic allowed by default [ 0.845383] vgaarb: loaded [ 0.847395] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.849015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.855287] clocksource: Switched to clocksource kvm-clock [ 0.973115] VFS: Disk quotas dquot_6.6.0 [ 0.974503] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.976727] *** VALIDATE ramfs *** [ 0.978082] *** VALIDATE hugetlbfs *** [ 0.979550] pnp: PnP ACPI init [ 0.982329] pnp: PnP ACPI: found 6 devices [ 0.997528] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.001015] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.003263] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.005511] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.008027] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.010381] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.013036] NET: Registered protocol family 2 [ 1.015623] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.021209] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.024788] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.031235] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.035271] TCP: Hash tables configured (established 65536 bind 65536) [ 1.038202] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.041417] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.044304] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.047776] NET: Registered protocol family 1 [ 1.051348] RPC: Registered named UNIX socket transport module. [ 1.053267] RPC: Registered udp transport module. [ 1.054624] RPC: Registered tcp transport module. [ 1.056634] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.059140] NET: Registered protocol family 44 [ 1.060886] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.063115] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.065403] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.067589] PCI: CLS 0 bytes, default 64 [ 1.069186] Unpacking initramfs... [ 2.612127] debug: unmapping init [mem 0xffff9ebf7cc54000-0xffff9ebf7ffbffff] [ 2.618630] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.620705] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.625883] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.219776] Initialise system trusted keyrings [ 3.223103] Key type blacklist registered [ 3.226037] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.237847] zbud: loaded [ 3.246880] *** VALIDATE nfs *** [ 3.248440] *** VALIDATE nfs4 *** [ 3.251015] pstore: using deflate compression [ 3.255134] Platform Keyring initialized [ 3.371725] NET: Registered protocol family 38 [ 3.373455] Key type asymmetric registered [ 3.374898] Asymmetric key parser 'x509' registered [ 3.376755] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.379550] io scheduler mq-deadline registered [ 3.381395] io scheduler kyber registered [ 3.382991] io scheduler bfq registered [ 3.384573] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.387530] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.390372] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.393125] ACPI: Power Button [PWRF] [ 3.398414] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.405626] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.421063] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.429499] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.451700] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.484300] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.516947] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.523387] Non-volatile memory driver v1.3 [ 3.525359] Linux agpgart interface v0.103 [ 3.558314] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.561100] vda: detected capacity change from 0 to 74694656 [ 3.578948] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.581419] vdb: detected capacity change from 0 to 1073741824 [ 3.600773] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.603734] vdc: detected capacity change from 0 to 2621440000 [ 3.619766] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.622840] vdd: detected capacity change from 0 to 2621440000 [ 3.638295] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.641692] vde: detected capacity change from 0 to 4294967296 [ 3.659197] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.662176] vdf: detected capacity change from 0 to 4294967296 [ 3.668766] libphy: Fixed MDIO Bus: probed [ 3.674314] usbcore: registered new interface driver usbserial_generic [ 3.677068] usbserial: USB Serial support registered for generic [ 3.679320] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.684243] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.686164] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.688583] mousedev: PS/2 mouse device common for all mice [ 3.692795] rtc_cmos 00:05: RTC can wake from S4 [ 3.697825] rtc_cmos 00:05: registered as rtc0 [ 3.699308] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.702964] intel_pstate: CPU model not supported [ 3.704732] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.712230] hid: raw HID events driver (C) Jiri Kosina [ 3.714199] usbcore: registered new interface driver usbhid [ 3.717110] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.718535] usbhid: USB HID core driver [ 3.718711] drop_monitor: Initializing network drop monitor service [ 3.718922] Initializing XFRM netlink socket [ 3.728851] NET: Registered protocol family 10 [ 3.731658] Segment Routing with IPv6 [ 3.733145] NET: Registered protocol family 17 [ 3.734943] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.736765] mpls_gso: MPLS GSO support [ 3.743997] RAS: Correctable Errors collector initialized. [ 3.746087] AVX version of gcm_enc/dec engaged. [ 3.747542] AES CTR mode by8 optimization enabled [ 3.819787] sched_clock: Marking stable (3819770677, 0)->(4858794831, -1039024154) [ 3.823399] registered taskstats version 1 [ 3.827465] Loading compiled-in X.509 certificates [ 3.829527] zswap: loaded using pool lzo/zbud [ 3.853720] Key type big_key registered [ 3.866196] Key type encrypted registered [ 3.867664] ima: No TPM chip found, activating TPM-bypass! [ 3.869624] ima: Allocated hash algorithm: sha1 [ 3.871401] ima: No architecture policies found [ 3.873272] evm: Initialising EVM extended attributes: [ 3.875312] evm: security.selinux [ 3.876480] evm: security.ima [ 3.877587] evm: security.capability [ 3.878973] evm: HMAC attrs: 0x1 [ 3.881405] rtc_cmos 00:05: setting system clock to 2026-08-11 16:01:27 UTC (1786464087) [ 3.888571] debug: unmapping init [mem 0xffffffff95e03000-0xffffffff95ffffff] [ 3.891525] debug: unmapping init [mem 0xffffffff94b82000-0xffffffff94e58fff] [ 3.901137] Write protecting the kernel read-only data: 28672k [ 3.905699] debug: unmapping init [mem 0xffffffff93203000-0xffffffff933fffff] [ 3.909171] debug: unmapping init [mem 0xffffffff93b14000-0xffffffff93bfffff] [ 3.952429] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.962403] systemd[1]: Detected virtualization kvm. [ 3.964523] systemd[1]: Detected architecture x86-64. [ 3.966617] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.996964] systemd[1]: No hostname configured. [ 4.000762] systemd[1]: Set hostname to . [ 4.003202] random: systemd: uninitialized urandom read (16 bytes read) [ 4.005736] systemd[1]: Initializing machine ID from random generator. [ 4.204559] random: systemd: uninitialized urandom read (16 bytes read) [ 4.207528] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.219854] random: systemd: uninitialized urandom read (16 bytes read) [ 4.222450] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.232228] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.168983] device-mapper: uevent: version 1.0.3 [ 5.171633] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Mounting Kernel Configuration File System... Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.368354] virtio_net virtio0 ens2: renamed from eth0 [ 6.426087] random: fast init done [ 7.360657] scsi host0: ata_piix [ 7.435452] scsi host1: ata_piix [ 7.437450] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.439658] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.388736] random: crng init done [ 12.391105] random: 7 urandom warning(s) missed due to ratelimiting [ 12.500462] dracut-initqueue[567]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.845280] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.399547] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.776733] SELinux: Disabled at runtime. [ 15.839793] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.850889] systemd[1]: Detected virtualization kvm. [ 15.852863] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.668597] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.672430] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.683301] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.688365] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.692423] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.702277] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.710584] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 17.117191] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.766597] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.947366] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.976229] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.790432] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.116107] EDAC sbridge: Ver: 1.1.2 [ 22.651511] Key type dns_resolver registered [ 23.058268] NFS: Registering the id_resolver key type [ 23.060285] Key type id_resolver registered [ 23.061744] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-server login: [ 65.400616] libcfs: loading out-of-tree module taints kernel. [ 65.426938] Key type ._llcrypt registered [ 65.428885] Key type .llcrypt registered [ 65.481858] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 74.496409] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 75.133781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.139780] alg: No test for adler32 (adler32-zlib) [ 76.162138] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 76.502151] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 78.135192] Key type lgssc registered [ 78.832218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.639864] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 104.644414] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 110.665524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 110.682653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.797598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 111.816911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 111.873199] Lustre: lustre-MDT0000: new disk, initializing [ 111.917974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 111.930457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 113.973794] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.637376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 120.700183] Lustre: 6463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 120.720365] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 120.723623] Lustre: Skipped 1 previous similar message [ 120.776959] Lustre: lustre-MDT0001: new disk, initializing [ 120.824458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 120.841769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 120.846459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 122.745177] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.570805] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 129.702328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 129.826972] Lustre: lustre-OST0000: new disk, initializing [ 129.829516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 129.833457] Lustre: 8401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 129.867854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 132.577357] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.182040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 136.186083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 136.220332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 140.048954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 140.108407] Lustre: lustre-OST0001: new disk, initializing [ 140.110927] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 140.114076] Lustre: 9471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 140.146933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 143.203895] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.375628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 144.382325] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 144.417281] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 151.375756] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 159.101407] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 162.028774] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing check_logdir /tmp/testlogs/ [ 164.070069] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing yml_node [ 166.087235] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 167.245681] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 168.396842] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 169.155459] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 11 12:04:12 EDT 2026 [ 176.975532] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 177.666353] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 190.431907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.432412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 190.437712] Lustre: Skipped 2 previous similar messages [ 190.441021] Lustre: Skipped 1 previous similar message [ 195.551995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 195.556049] Lustre: Skipped 3 previous similar messages [ 200.671968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 200.675298] Lustre: Skipped 3 previous similar messages [ 204.255149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 204.322943] Lustre: server umount lustre-MDT0000 complete [ 205.838197] LustreError: 6455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464289 with bad export cookie 9891380233987091164 [ 205.841261] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.843025] LustreError: 6455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 210.912598] LustreError: 10049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 210.913129] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.913569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 210.918847] LustreError: 10049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 210.934019] Lustre: Skipped 2 previous similar messages [ 216.031871] LustreError: 6471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 216.031881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 216.031888] Lustre: Skipped 1 previous similar message [ 216.040665] LustreError: 6471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 217.055730] LustreError: 10049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.063782] LustreError: 10049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 220.127528] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 220.225381] Lustre: server umount lustre-MDT0001 complete [ 222.057855] Lustre: server umount lustre-OST0000 complete [ 223.906531] Lustre: server umount lustre-OST0001 complete [ 226.805734] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 231.428990] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 248.059612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.174144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 248.191421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 248.236336] Lustre: lustre-MDT0000: new disk, initializing [ 248.277223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.289031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 250.233586] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 255.858314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 255.914618] Lustre: 15011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 256.056317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 257.983981] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 262.902715] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 265.322409] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 268.301685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.443924] Lustre: lustre-OST0000: new disk, initializing [ 268.446609] Lustre: Skipped 1 previous similar message [ 268.450054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 268.454825] Lustre: Skipped 2 previous similar messages [ 268.459286] Lustre: 16981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.509800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 268.513047] Lustre: Skipped 1 previous similar message [ 270.152669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.158537] Lustre: Skipped 1 previous similar message [ 270.161537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.211968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 271.127450] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 276.503374] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 278.516617] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 278.608584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 280.445570] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 280.557636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 285.665706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 285.667639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.670543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 285.680844] Lustre: Skipped 1 previous similar message [ 285.686690] Lustre: Skipped 2 previous similar messages [ 287.512313] Lustre: server umount lustre-OST0000 complete [ 290.785778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.793564] Lustre: Skipped 1 previous similar message [ 295.279765] Lustre: server umount lustre-MDT0000 complete [ 295.905515] LustreError: 15018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.912965] LustreError: 15018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 296.926560] LustreError: 15004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464380 with bad export cookie 9891380233987092697 [ 296.930054] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.932222] LustreError: 15004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 297.061281] Lustre: server umount lustre-MDT0001 complete [ 298.907855] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:06:21 (1786464381) [ 299.978623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 300.758764] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:06:23 (1786464383) [ 303.744524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.963320] LustreError: 19387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.008648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 305.862069] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 309.948349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 312.235293] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 315.862548] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 318.395583] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 321.567970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 324.441597] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 328.143164] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 343.030866] Lustre: server umount lustre-OST0000 complete [ 345.570554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.571515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 345.579079] Lustre: Skipped 1 previous similar message [ 345.581720] Lustre: Skipped 3 previous similar messages [ 354.819535] Lustre: server umount lustre-MDT0000 complete [ 355.807945] LustreError: 19406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.815243] LustreError: 19406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 356.551129] LustreError: 19369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464440 with bad export cookie 9891380233987093579 [ 356.554664] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.560057] LustreError: 19369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 367.143621] Lustre: server umount lustre-MDT0001 complete [ 372.330460] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 373.584679] Key type lgssc unregistered [ 373.739461] LNet: 22942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.743361] LNetError: 22942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 373.753519] LNet: Removed LNI 192.168.203.106@tcp [ 374.166986] Key type .llcrypt unregistered [ 374.168908] Key type ._llcrypt unregistered [ 385.923175] Key type ._llcrypt registered [ 385.924724] Key type .llcrypt registered [ 385.976290] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 386.374632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 386.439702] alg: No test for adler32 (adler32-zlib) [ 387.330385] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 387.449345] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 389.047163] Key type lgssc registered [ 389.506524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 393.009861] loop: module loaded [ 393.037479] loop0: detected capacity change from 0 to 204800000 [ 393.114411] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 393.119684] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 393.266222] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 395.879446] loop0: detected capacity change from 0 to 204800000 [ 395.961258] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 395.970293] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 395.975937] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 398.318694] loop0: detected capacity change from 0 to 204800000 [ 400.768659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 400.777343] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.879744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.899842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.936865] Lustre: lustre-MDT0000: new disk, initializing [ 401.969603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.981283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.718116] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 408.237990] loop1: detected capacity change from 0 to 204800000 [ 410.621452] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 410.755542] Lustre: lustre-OST07c6: new disk, initializing [ 410.757744] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 410.760204] Lustre: Skipped 1 previous similar message [ 410.763289] Lustre: 26106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 410.802534] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 411.404511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 411.409932] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 411.425129] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 413.303355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 416.769742] Lustre: 26114:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 416.774578] Lustre: 26114:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 416.778132] Lustre: 26114:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 416.781658] Lustre: 26114:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 416.785476] Lustre: 26114:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 416.789150] Lustre: 26114:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 432.617835] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 432.622777] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 432.626856] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 432.630234] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 432.634406] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 432.640320] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 436.375758] Lustre: 26114:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 436.380966] Lustre: 26114:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 436.384375] Lustre: 26114:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 436.389246] Lustre: 26114:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.393444] Lustre: 26114:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 436.397904] Lustre: 26114:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.402105] Lustre: 26114:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 436.407273] Lustre: 26114:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.411130] Lustre: 26114:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 436.414467] Lustre: 26114:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 436.417819] Lustre: 26114:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 436.421457] Lustre: 26114:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.505359] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 439.510905] Lustre: 25093:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 439.514572] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 439.519258] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.523152] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 439.527193] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.531440] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 439.535563] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.539433] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 439.542191] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 439.545377] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 439.548938] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.421311] Lustre: 25092:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 446.426458] Lustre: 25092:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 446.430181] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 446.433853] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.439698] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 446.444717] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.449366] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 446.454114] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.458354] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 446.461906] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 446.465604] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 446.470521] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 455.747825] Lustre: 25092:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 455.752295] Lustre: 25092:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 455.755593] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 455.758932] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 455.762890] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 455.766099] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 455.769482] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 455.773300] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 455.776904] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 455.780732] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 455.784215] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 455.787583] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 498.656988] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 498.657551] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.665775] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 499.284450] Lustre: server umount lustre-OST07c6 complete [ 503.176031] Lustre: server umount lustre-MDT0000 complete [ 509.508467] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 510.654504] Key type lgssc unregistered [ 510.794386] LNet: 28682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.797439] LNetError: 28682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.805836] LNet: Removed LNI 192.168.203.106@tcp [ 511.146860] Key type .llcrypt unregistered [ 511.148366] Key type ._llcrypt unregistered [ 519.571510] Key type ._llcrypt registered [ 519.573145] Key type .llcrypt registered [ 519.618339] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 519.984471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 520.030067] alg: No test for adler32 (adler32-zlib) [ 520.909940] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 521.015964] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 522.607154] Key type lgssc registered [ 523.077476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 526.322739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 526.331929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.525142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.251568] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 532.652538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.805278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 534.538323] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 537.669352] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.065085] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 542.707238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 542.862567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.091453] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 548.055642] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 553.132060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 562.148328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 562.154597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.162624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.687615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 563.688189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.726772] Lustre: server umount lustre-OST0000 complete [ 568.800521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.807655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 568.812608] Lustre: Skipped 1 previous similar message [ 568.813320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 568.816799] Lustre: Skipped 2 previous similar messages [ 568.824497] LustreError: Skipped 1 previous similar message [ 573.925080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 573.947198] Lustre: Skipped 1 previous similar message [ 579.042265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 582.115046] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 582.231510] Lustre: server umount lustre-MDT0000 complete [ 584.160629] LustreError: 29924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.169666] LustreError: 29924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 584.822986] LustreError: 29910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464668 with bad export cookie 9932193892277389389 [ 584.831449] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.835266] LustreError: 29910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 585.083872] Lustre: server umount lustre-MDT0001 complete [ 593.455709] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 595.326246] Key type lgssc unregistered [ 595.513697] LNet: 33662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.523709] LNetError: 33662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.581701] LNet: Removed LNI 192.168.203.106@tcp [ 597.191995] Key type .llcrypt unregistered [ 597.194353] Key type ._llcrypt unregistered [ 612.292245] Key type ._llcrypt registered [ 612.294210] Key type .llcrypt registered [ 612.354254] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 613.061381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.141626] alg: No test for adler32 (adler32-zlib) [ 614.101574] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 614.286420] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 615.951190] Key type lgssc registered [ 616.690461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 622.181573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 622.193433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.424121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.918284] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 631.557225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.787204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 635.008819] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 639.735280] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 643.317420] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 647.677680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 647.869365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.944510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 652.272582] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 656.700421] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 664.545940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 664.549976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.557100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 667.618056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 667.618065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.620050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 672.741022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 672.745757] Lustre: Skipped 2 previous similar messages [ 677.352500] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 677.428323] Lustre: server umount lustre-OST0000 complete [ 682.979021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.979221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.986209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 682.996291] Lustre: Skipped 1 previous similar message [ 688.097036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.100918] Lustre: Skipped 1 previous similar message [ 694.239694] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 694.321455] Lustre: server umount lustre-MDT0000 complete [ 696.471483] LustreError: 35645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464780 with bad export cookie 9580002848207201491 [ 696.475080] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.485597] LustreError: 35645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 696.715834] Lustre: server umount lustre-MDT0001 complete [ 702.384596] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 703.819228] Key type lgssc unregistered [ 704.004668] LNet: 38448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.008381] LNetError: 38448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.021745] LNet: Removed LNI 192.168.203.106@tcp [ 704.503331] Key type .llcrypt unregistered [ 704.506196] Key type ._llcrypt unregistered [ 717.273440] Key type ._llcrypt registered [ 717.275956] Key type .llcrypt registered [ 717.332642] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 717.777201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.783649] alg: No test for adler32 (adler32-zlib) [ 718.695652] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 718.868097] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 720.479198] Key type lgssc registered [ 721.203707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 725.691960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 725.710588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.958135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.052280] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 733.714923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.934385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 736.314844] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 740.555900] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 744.362911] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 748.619336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 748.838638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.936874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 753.061813] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 758.380639] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 764.427754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.429472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.434436] Lustre: Skipped 2 previous similar messages [ 764.444392] Lustre: Skipped 2 previous similar messages [ 765.846501] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 766.244455] Lustre: server umount lustre-MDT0000 complete [ 768.887840] LustreError: 39824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464852 with bad export cookie 2280371602624580817 [ 768.891679] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.904470] LustreError: 39824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 769.504697] LustreError: 42332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 769.505679] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.527772] LustreError: 42332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 769.538512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 770.934202] LustreError: 39841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.538269] LustreError: 42420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 773.540388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 773.552285] Lustre: Skipped 1 previous similar message [ 776.056812] LustreError: 39839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 778.656705] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 778.660496] Lustre: Skipped 1 previous similar message [ 781.174435] LustreError: 39841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.187790] LustreError: 39841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 783.327124] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 783.474228] Lustre: server umount lustre-MDT0001 complete [ 802.733302] Lustre: server umount lustre-OST0000 complete [ 812.946284] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 814.474419] Key type lgssc unregistered [ 814.644692] LNet: 43703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.649994] LNetError: 43703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.657443] LNet: Removed LNI 192.168.203.106@tcp [ 815.105932] Key type .llcrypt unregistered [ 815.108107] Key type ._llcrypt unregistered [ 830.309897] Key type ._llcrypt registered [ 830.311890] Key type .llcrypt registered [ 830.386660] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 830.932846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.943860] alg: No test for adler32 (adler32-zlib) [ 831.838919] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 831.975889] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 833.583161] Key type lgssc registered [ 834.191643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.512527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.524562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.750787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.803195] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 846.401751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.588773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 849.001234] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 853.059569] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 856.212777] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 859.390038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.554704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.035455] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 864.681938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 865.466145] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.099491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.104740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.112906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.058538] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 882.146700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.153218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.159411] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 887.163644] Lustre: Skipped 2 previous similar messages [ 892.278620] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 892.282219] Lustre: Skipped 2 previous similar messages [ 892.383110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.448698] Lustre: server umount lustre-OST0000 complete [ 907.744961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.745756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.752678] Lustre: Skipped 1 previous similar message [ 907.755865] Lustre: Skipped 2 previous similar messages [ 912.175783] Lustre: server umount lustre-MDT0000 complete [ 912.864173] LustreError: 45096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 912.873768] LustreError: 45096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 913.903482] LustreError: 45078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786464997 with bad export cookie 9956732603888770159 [ 913.906730] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.910961] LustreError: 45078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 928.223174] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 928.317488] Lustre: server umount lustre-MDT0001 complete [ 933.717493] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 935.232986] Key type lgssc unregistered [ 935.388797] LNet: 48825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.392357] LNetError: 48825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.402900] LNet: Removed LNI 192.168.203.106@tcp [ 935.845193] Key type .llcrypt unregistered [ 935.848515] Key type ._llcrypt unregistered [ 948.187132] Key type ._llcrypt registered [ 948.188661] Key type .llcrypt registered [ 948.232944] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 948.660304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.672386] alg: No test for adler32 (adler32-zlib) [ 949.555780] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 949.655048] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 951.255225] Key type lgssc registered [ 951.768922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 955.108719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 955.116652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.320336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.138855] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 962.147957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.339442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 964.181946] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 967.694571] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 970.325794] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 973.489964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.668813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.301942] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 979.585534] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 980.838976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 992.047159] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 993.640561] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1004.517889] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1112h18m07s [ 1005.158167] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1006.461919] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1007.477971] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1007.585120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.591281] Lustre: Skipped 2 previous similar messages [ 1008.609784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.616199] Lustre: Skipped 5 previous similar messages [ 1013.621799] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1018.677702] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1018.681524] Lustre: Skipped 4 previous similar messages [ 1020.383131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1020.468922] Lustre: server umount lustre-MDT0000 complete [ 1020.897179] LustreError: 51318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.902659] LustreError: 51318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1021.928345] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1024.714836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.795214] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.904471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.931301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1025.957174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1026.726912] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1028.318466] Lustre: DEBUG MARKER: Wait for df (36732) ... [ 1029.059784] Lustre: DEBUG MARKER: done [ 1030.454109] Lustre: server umount lustre-OST0000 complete [ 1036.257112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.257997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.258065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1036.264447] Lustre: Skipped 1 previous similar message [ 1036.268473] Lustre: Skipped 1 previous similar message [ 1041.474261] Lustre: server umount lustre-MDT0000 complete [ 1043.085286] LustreError: 50201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465126 with bad export cookie 664796953404680573 [ 1043.088782] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1043.090962] LustreError: 50201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1052.696381] Lustre: server umount lustre-MDT0001 complete [ 1057.035355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1058.210438] Key type lgssc unregistered [ 1058.353609] LNet: 55162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.357374] LNetError: 55162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.366593] LNet: Removed LNI 192.168.203.106@tcp [ 1058.740431] Key type .llcrypt unregistered [ 1058.742031] Key type ._llcrypt unregistered [ 1076.129618] Key type ._llcrypt registered [ 1076.131919] Key type .llcrypt registered [ 1076.175922] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1076.598931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.664416] alg: No test for adler32 (adler32-zlib) [ 1077.559572] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1077.669240] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1079.271161] Key type lgssc registered [ 1079.750541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1083.050239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1083.059258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.174485] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1084.185220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1084.251947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.947745] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1089.684519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.745900] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1089.759457] Lustre: 57346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1089.889666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.676556] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1095.055055] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1097.536867] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1100.632375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1100.728982] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1100.831080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.071327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1103.400673] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1106.503774] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1110.145289] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1112h19m53s [ 1110.734744] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1113.406254] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1114.315241] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1114.334996] LustreError: 57355:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1134.470299] Lustre: lustre-MDT0000: Client df9b4104-e418-45d8-8bcf-9df116b9d15f (at 192.168.203.6@tcp) reconnecting [ 1154.383142] LustreError: 57355:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1155.185528] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1156.087709] Lustre: DEBUG MARKER: done [ 1160.161803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1160.164335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.168943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.721909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1162.722340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1162.723315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.725398] Lustre: Skipped 1 previous similar message [ 1164.053105] Lustre: server umount lustre-OST0000 complete [ 1166.815964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1166.820410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.825045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.565694] Lustre: server umount lustre-MDT0000 complete [ 1172.951275] LustreError: 60104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465256 with bad export cookie 1678813299906932869 [ 1172.954754] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1172.956646] LustreError: 60104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1173.078332] Lustre: server umount lustre-MDT0001 complete [ 1177.015202] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1178.181694] Key type lgssc unregistered [ 1178.332438] LNet: 61671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.335796] LNetError: 61671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.344768] LNet: Removed LNI 192.168.203.106@tcp [ 1178.705198] Key type .llcrypt unregistered [ 1178.706923] Key type ._llcrypt unregistered [ 1195.707838] Key type ._llcrypt registered [ 1195.709431] Key type .llcrypt registered [ 1195.746952] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1196.066612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1196.120080] alg: No test for adler32 (adler32-zlib) [ 1196.977796] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1197.079512] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1198.663158] Key type lgssc registered [ 1199.050270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1200.260729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1200.267467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1201.376325] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1201.386704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1201.451901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1201.805496] Lustre: Failing over lustre-MDT0000 [ 1201.910303] Lustre: server umount lustre-MDT0000 complete [ 1204.232810] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:21:27 (1786465287) [ 1206.614684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.777688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.302759] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1211.225909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.266483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1211.277863] Lustre: 64443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1211.374922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1212.738648] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1215.131807] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.948333] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1219.067710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1219.133092] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1219.192389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.065509] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1222.923756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1223.405684] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1226.435895] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1228.258261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.259123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1228.263019] Lustre: Skipped 2 previous similar messages [ 1228.266024] Lustre: Skipped 1 previous similar message [ 1233.377708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.380265] Lustre: Skipped 2 previous similar messages [ 1234.331403] Lustre: server umount lustre-MDT0000 complete [ 1235.636937] LustreError: 66298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465319 with bad export cookie 15833141315696206309 [ 1235.640409] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.642583] LustreError: 66298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1235.741775] Lustre: server umount lustre-MDT0001 complete [ 1236.835319] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1238.406649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1241.211556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.663604] LustreError: 63368:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c9054bb7 x1873244617472256/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1245.800951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.830083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1247.184664] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1250.003353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.095839] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.482031] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1253.843838] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1254.880088] Lustre: 63372:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786465322/real 1786465322] req@000000009d9e82ed x1873244617471872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786465338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1255.398039] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1255.670952] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1256.416101] Lustre: 63369:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786465324/real 1786465324] req@00000000c6ff755c x1873244617472000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786465340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1259.360805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1259.362111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.363637] Lustre: Skipped 1 previous similar message [ 1259.369227] Lustre: Skipped 1 previous similar message [ 1260.383151] Lustre: 63370:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786465327/real 1786465327] req@000000005749b908 x1873244617472128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786465343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1263.440948] Lustre: server umount lustre-MDT0000 complete [ 1264.579939] LustreError: 64437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465348 with bad export cookie 15833141315696208577 [ 1264.583608] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.583824] LustreError: 64437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1264.607633] LustreError: 68207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.608855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.614105] LustreError: 68207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1264.620482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1264.623114] Lustre: Skipped 1 previous similar message [ 1264.688480] Lustre: server umount lustre-MDT0001 complete [ 1266.686226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1269.503736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.847513] LustreError: 63368:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b71c7b79 x1873244617490560/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1274.951075] LustreError: 68228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.974535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.977454] Lustre: Skipped 1 previous similar message [ 1274.999805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1276.213825] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1278.859000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.235756] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1282.377528] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1284.064916] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1285.092308] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1289.184345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1289.187288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.191472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.794173] Lustre: server umount lustre-MDT0000 complete [ 1292.962109] LustreError: 64437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465376 with bad export cookie 15833141315696209949 [ 1292.964203] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1292.966648] LustreError: 64437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1293.068274] Lustre: server umount lustre-MDT0001 complete [ 1295.117747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1296.209723] Lustre: DEBUG MARKER: files compared the same [ 1307.166749] Lustre: server umount lustre-OST0000 complete [ 1312.553920] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1313.607294] Key type lgssc unregistered [ 1313.735801] LNet: 74158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.739200] LNetError: 74158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.750292] LNet: Removed LNI 192.168.203.106@tcp [ 1314.058615] Key type .llcrypt unregistered [ 1314.060057] Key type ._llcrypt unregistered [ 1322.855496] Key type ._llcrypt registered [ 1322.856684] Key type .llcrypt registered [ 1322.892688] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1323.184355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1323.193506] alg: No test for adler32 (adler32-zlib) [ 1324.038108] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1324.119741] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1325.703167] Key type lgssc registered [ 1326.066056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1328.638118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1328.644548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.795540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.098644] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 1333.851704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.943706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.280417] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 1337.636428] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1339.374852] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1341.505419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1341.602464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1343.415161] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug +malloc all [ 1345.766072] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1355.939540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1361.375782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.376796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.376974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1361.381212] Lustre: Skipped 1 previous similar message [ 1365.780159] Lustre: server umount lustre-OST0000 complete [ 1370.079507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.082739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.087882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.089673] Lustre: Skipped 1 previous similar message [ 1371.615683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.615830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.735737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.738306] Lustre: Skipped 2 previous similar messages [ 1381.343182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1381.391530] Lustre: server umount lustre-MDT0000 complete [ 1381.855671] LustreError: 75550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.862225] LustreError: 75550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1382.587704] LustreError: 78048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465466 with bad export cookie 14889233341153207164 [ 1382.589390] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1382.591514] LustreError: 78048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1382.691239] Lustre: server umount lustre-MDT0001 complete [ 1386.077437] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1387.107957] Key type lgssc unregistered [ 1387.242351] LNet: 79107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.245142] LNetError: 79107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.254367] LNet: Removed LNI 192.168.203.106@tcp [ 1387.553903] Key type .llcrypt unregistered [ 1387.555442] Key type ._llcrypt unregistered [ 1396.252450] Key type ._llcrypt registered [ 1396.253817] Key type .llcrypt registered [ 1396.289151] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1396.591322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.621026] alg: No test for adler32 (adler32-zlib) [ 1397.470498] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1397.563633] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1399.151123] Key type lgssc registered [ 1399.517875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1402.109319] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1402.115825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.567506] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1422.687262] LustreError: 80474:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 1449.311142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.213345] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1453.606929] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1456.079306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.113172] Lustre: *** cfs_fail_loc=706, val=0*** [ 1456.233146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.253657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1457.484900] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1460.258972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.381890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.621163] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1463.932453] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1465.819601] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1466.847907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.851089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.856635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.897237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.899094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.650641] Lustre: server umount lustre-OST0000 complete [ 1474.016475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.017224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.020319] Lustre: Skipped 1 previous similar message [ 1474.022078] Lustre: Skipped 1 previous similar message [ 1479.136051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.138053] Lustre: Skipped 2 previous similar messages [ 1484.255884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1484.258028] Lustre: Skipped 1 previous similar message [ 1488.351112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1488.408931] Lustre: server umount lustre-MDT0000 complete [ 1489.376046] LustreError: 81561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.382281] LustreError: 81561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1489.617122] LustreError: 83130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465573 with bad export cookie 13288227516340165574 [ 1489.620676] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1489.621771] LustreError: 83130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1489.729472] Lustre: server umount lustre-MDT0001 complete [ 1493.035863] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1494.032785] Key type lgssc unregistered [ 1494.165365] LNet: 84086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.167479] LNetError: 84086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.176299] LNet: Removed LNI 192.168.203.106@tcp [ 1494.479848] Key type .llcrypt unregistered [ 1494.480800] Key type ._llcrypt unregistered [ 1503.663247] Key type ._llcrypt registered [ 1503.664621] Key type .llcrypt registered [ 1503.721492] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1504.184082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1504.199720] alg: No test for adler32 (adler32-zlib) [ 1505.062902] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1505.158175] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1506.743171] Key type lgssc registered [ 1507.070494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1509.544843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1509.550526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.794593] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1514.239519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.405306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.622349] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1518.127868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.225136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.073604] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1522.777995] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1522.784704] Lustre: MGS: Client 9e2c2f8c-3c8c-4654-b75d-77300df4ac6f (at 0@lo) reconnecting [ 1522.792168] LustreError: 86179:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1522.862800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.888970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1524.186969] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1533.408181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1533.411352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.416650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.435605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.435666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.437318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.444821] Lustre: Skipped 1 previous similar message [ 1538.060661] Lustre: server umount lustre-OST0000 complete [ 1539.552467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.552501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1539.553255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1539.556039] Lustre: Skipped 1 previous similar message [ 1544.672735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.676336] Lustre: Skipped 2 previous similar messages [ 1545.409643] Lustre: server umount lustre-MDT0000 complete [ 1546.704776] Lustre: server umount lustre-MDT0001 complete [ 1547.989979] Lustre: server umount MGS complete [ 1551.702769] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:27:14 (1786465634) [ 1560.635621] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 1563.205469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1574.297250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.723382] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1581.789116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.970203] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1582.001377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1582.060256] Lustre: lustre-MDT0001: new disk, initializing [ 1582.110047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.355452] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1582.366672] Lustre: 93321:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1582.374715] LustreError: 93321:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1585.785686] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1593.357112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1593.495832] Lustre: lustre-OST0000: new disk, initializing [ 1593.498608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1593.572902] Lustre: 94086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1593.629378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.985231] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1604.640335] Lustre: 94738:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000029eaab43 x1873244940436608/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786465704 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1605.391133] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1609.653360] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1609.661706] Lustre: MGS: Client fc05b845-b3ac-4b13-b0ab-51aeb7b8e03f (at 0@lo) reconnecting [ 1609.692553] Lustre: 92591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1609.705631] LustreError: 93307:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1609.727529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1609.778826] Lustre: lustre-MDT0000: new disk, initializing [ 1609.885452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.903157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1610.976177] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1610.977023] Lustre: 95145:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ff807216 x1873244940442496/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786465710 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1610.997247] Lustre: 95145:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 1613.036771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1613.042483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1613.067329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1613.546594] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1634.273181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.280756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.290119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.354650] Lustre: server umount lustre-OST0000 complete [ 1642.977340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.977398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1642.983437] Lustre: Skipped 1 previous similar message [ 1642.987690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1642.987699] Lustre: Skipped 3 previous similar messages [ 1642.999841] LustreError: Skipped 1 previous similar message [ 1648.464522] Lustre: server umount lustre-MDT0000 complete [ 1656.659933] Lustre: server umount lustre-MDT0001 complete [ 1658.463585] Lustre: server umount MGS complete [ 1664.075924] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:29:06 (1786465746) [ 1672.573167] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1674.066778] Key type lgssc unregistered [ 1674.266887] LNet: 97652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.271559] LNetError: 97652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.282892] LNet: Removed LNI 192.168.203.106@tcp [ 1674.773735] Key type .llcrypt unregistered [ 1674.775751] Key type ._llcrypt unregistered [ 1684.130083] Key type ._llcrypt registered [ 1684.131838] Key type .llcrypt registered [ 1684.184653] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1684.674971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1684.744132] alg: No test for adler32 (adler32-zlib) [ 1685.643324] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1685.774791] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1687.383181] Key type lgssc registered [ 1687.934634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.896170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.911231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.166921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.210715] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1702.387872] LustreError: 98852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.420138] LustreError: 98852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1703.724617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1703.936643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1707.373774] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1714.783866] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1720.022664] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1725.767070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1726.083345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.123540] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1738.114301] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1738.354975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1748.960140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1748.980641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.987740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1749.004516] Lustre: Skipped 1 previous similar message [ 1749.017052] Lustre: Skipped 1 previous similar message [ 1752.487887] Lustre: server umount lustre-OST0000 complete [ 1759.202143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.208963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.256616] Lustre: Skipped 1 previous similar message [ 1759.267040] Lustre: Skipped 1 previous similar message [ 1761.566797] Lustre: server umount lustre-MDT0000 complete [ 1764.324658] LustreError: 100490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.338138] LustreError: 100490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1764.349596] LustreError: 98831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465847 with bad export cookie 4925858296017426600 [ 1764.359089] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.362472] LustreError: 98831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1764.612230] Lustre: server umount lustre-MDT0001 complete [ 1772.807939] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1774.754928] Key type lgssc unregistered [ 1774.940499] LNet: 102388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.953607] LNetError: 102388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.969470] LNet: Removed LNI 192.168.203.106@tcp [ 1775.605813] Key type .llcrypt unregistered [ 1775.607096] Key type ._llcrypt unregistered [ 1787.481842] Key type ._llcrypt registered [ 1787.483526] Key type .llcrypt registered [ 1787.570515] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1788.149427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.166439] alg: No test for adler32 (adler32-zlib) [ 1789.138302] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1789.304046] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1790.943202] Key type lgssc registered [ 1791.673601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.754391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1795.757650] LustreError: 103448:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1795.766835] LustreError: 103450:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1795.774611] LustreError: 103448:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 1795.782516] LustreError: 103448:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1795.790920] LustreError: 103448:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1795.795307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.108268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.827241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.973262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.547355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1807.725689] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing lsmod [ 1809.523958] LustreError: 104649:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1809.544244] LustreError: 104655:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1809.550885] LustreError: 104649:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 1809.557639] LustreError: 104649:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1809.561779] LustreError: 104649:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1809.569596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1809.859917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.448311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1812.455235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1812.468663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1813.477980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1813.480695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.502589] Lustre: Skipped 1 previous similar message [ 1817.847262] Lustre: server umount lustre-MDT0000 complete [ 1818.597083] LustreError: 104841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1818.630080] LustreError: 104841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1820.146767] LustreError: 103464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465903 with bad export cookie 3818411357643814340 [ 1820.151475] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1820.154286] LustreError: 103464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1821.663439] LustreError: 103918:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c8f7e8cb x1873245238215936/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1821.744659] Lustre: server umount lustre-MDT0001 complete [ 1833.530709] Lustre: server umount lustre-OST0000 complete [ 1839.164042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.429471] LustreError: 105717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1839.486601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.617416] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1844.716428] LustreError: 105718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.731135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.026808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1851.985660] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1857.514598] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.605919] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.267861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.464240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.746531] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1875.459668] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1884.924793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1889.248548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1889.252113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.261145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1889.269866] Lustre: Skipped 2 previous similar messages [ 1891.327096] Lustre: server umount lustre-OST0000 complete [ 1892.281528] hrtimer: interrupt took 2608613 ns [ 1895.393468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.393697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1895.399908] Lustre: Skipped 1 previous similar message [ 1895.400922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.413760] Lustre: Skipped 3 previous similar messages [ 1900.512334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1900.526701] Lustre: Skipped 1 previous similar message [ 1908.191114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1908.268786] Lustre: server umount lustre-MDT0000 complete [ 1910.662201] LustreError: 105699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786465994 with bad export cookie 3818411357643815026 [ 1910.667663] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.669178] LustreError: 105699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1910.854428] Lustre: server umount lustre-MDT0001 complete [ 1917.718179] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1919.837817] Key type lgssc unregistered [ 1920.134710] LNet: 109273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.139129] LNetError: 109273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.147758] LNet: Removed LNI 192.168.203.106@tcp [ 1920.776913] Key type .llcrypt unregistered [ 1920.778791] Key type ._llcrypt unregistered [ 1936.887545] Key type ._llcrypt registered [ 1936.889164] Key type .llcrypt registered [ 1936.958333] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1937.564171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.579299] alg: No test for adler32 (adler32-zlib) [ 1938.517214] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1938.664198] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1940.303923] Key type lgssc registered [ 1941.088977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.028750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1946.040183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.296260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.363652] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1956.422625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.658945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1960.175768] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1965.182075] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1969.014547] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1973.237673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1973.458257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.746346] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1982.742499] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1992.932530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1996.683317] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1999.047144] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2002.401657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.407048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.419666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.434342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.435954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.444377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.464544] Lustre: Skipped 1 previous similar message [ 2007.541557] Lustre: server umount lustre-OST0000 complete [ 2013.152656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2013.158639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.174313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.019508] Lustre: server umount lustre-MDT0000 complete [ 2018.787162] LustreError: 110666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.801732] LustreError: 110666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2019.004433] LustreError: 110647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466102 with bad export cookie 11063033462032380626 [ 2019.007209] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.017921] LustreError: 110647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2019.236372] Lustre: server umount lustre-MDT0001 complete [ 2026.935839] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2028.689526] Key type lgssc unregistered [ 2028.886750] LNet: 114324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.891367] LNetError: 114324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.902879] LNet: Removed LNI 192.168.203.106@tcp [ 2029.461128] Key type .llcrypt unregistered [ 2029.463317] Key type ._llcrypt unregistered [ 2041.948909] Key type ._llcrypt registered [ 2041.950595] Key type .llcrypt registered [ 2042.033966] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2042.756559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2042.778415] alg: No test for adler32 (adler32-zlib) [ 2043.799534] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 2044.028226] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2045.691957] Key type lgssc registered [ 2046.448969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2051.971799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2051.984620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.400892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.554987] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2062.267866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.486384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.216822] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2070.020784] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.891201] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2078.659720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2078.871180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.182848] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2084.335381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2088.551338] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2094.560943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2094.570455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.578840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.632496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.634902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2097.637278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.262268] Lustre: server umount lustre-OST0000 complete [ 2102.759677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.776582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.779473] Lustre: Skipped 1 previous similar message [ 2107.880230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.886087] Lustre: Skipped 1 previous similar message [ 2112.994886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.999506] Lustre: Skipped 1 previous similar message [ 2117.087633] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2117.196155] Lustre: server umount lustre-MDT0000 complete [ 2118.117872] LustreError: 115521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.129994] LustreError: 115521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2119.574848] LustreError: 115506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466203 with bad export cookie 1193911655636756841 [ 2119.585074] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2119.585525] LustreError: 115506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2119.813946] Lustre: server umount lustre-MDT0001 complete [ 2126.143863] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2127.920809] Key type lgssc unregistered [ 2128.094638] LNet: 119056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.099996] LNetError: 119056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.115617] LNet: Removed LNI 192.168.203.106@tcp [ 2128.585410] Key type .llcrypt unregistered [ 2128.587656] Key type ._llcrypt unregistered [ 2142.581385] Key type ._llcrypt registered [ 2142.582878] Key type .llcrypt registered [ 2142.641478] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2143.131574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2143.192654] alg: No test for adler32 (adler32-zlib) [ 2144.130809] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 2144.276593] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2145.887208] Key type lgssc registered [ 2146.589811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2151.243914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.255273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.481471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.198099] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2160.842458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.065182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2164.261134] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2169.174462] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2173.010991] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2177.340116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2177.446954] LustreError: 122273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2177.460551] LustreError: 122273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2177.530347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.896686] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2182.643463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2185.227437] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2190.878431] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2193.151750] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2193.157809] Lustre: Skipped 2 previous similar messages [ 2196.093220] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2196.097278] Lustre: Skipped 2 previous similar messages [ 2197.010777] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2197.039465] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2205.268995] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2205.276468] Lustre: Skipped 2 previous similar messages [ 2211.355494] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.6@tcp 192.168.0.[2,10]@tcp [ 2213.767922] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.6/24@tcp in log lustre-MDT0000 [ 2213.772445] Lustre: Skipped 2 previous similar messages [ 2219.020966] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.6/24@tcp [ 2219.027281] Lustre: Skipped 1 previous similar message [ 2223.584807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.591680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.603054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.955984] Lustre: server umount lustre-OST0000 complete [ 2227.680179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2227.685438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.692397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.778464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.779211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.397897] Lustre: server umount lustre-MDT0000 complete [ 2235.313697] LustreError: 120437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466318 with bad export cookie 17937994154874936126 [ 2235.314323] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.321944] LustreError: 120437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2244.664370] Lustre: server umount lustre-MDT0001 complete [ 2250.729871] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2252.376383] Key type lgssc unregistered [ 2252.571869] LNet: 125436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.578832] LNetError: 125436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.588854] LNet: Removed LNI 192.168.203.106@tcp [ 2253.144824] Key type .llcrypt unregistered [ 2253.147772] Key type ._llcrypt unregistered [ 2271.102285] Key type ._llcrypt registered [ 2271.104644] Key type .llcrypt registered [ 2271.174534] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2271.674440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.683526] alg: No test for adler32 (adler32-zlib) [ 2272.630761] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 2272.807836] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2274.439119] Key type lgssc registered [ 2275.146044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2279.598852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2279.615812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.835446] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2280.842664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2280.847804] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.6/24@tcp [ 2283.257721] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2287.787606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.956586] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2287.974238] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.6/24@tcp [ 2288.013109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2290.423751] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2294.147957] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2297.224742] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2300.871431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.064867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.521643] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2306.220600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2308.715670] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2318.818140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2318.819767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.830756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2318.839344] Lustre: Skipped 1 previous similar message [ 2318.846811] Lustre: Skipped 1 previous similar message [ 2320.736782] Lustre: server umount lustre-OST0000 complete [ 2323.937354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.937636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2323.946123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2323.954727] Lustre: Skipped 1 previous similar message [ 2328.818929] Lustre: server umount lustre-MDT0000 complete [ 2329.056473] LustreError: 126971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.067398] LustreError: 126971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2330.974206] LustreError: 126956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466414 with bad export cookie 16822221580754324515 [ 2330.976399] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2330.981208] LustreError: 126956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.166839] Lustre: server umount lustre-MDT0001 complete [ 2337.142205] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2338.939627] Key type lgssc unregistered [ 2339.115451] LNet: 130567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.119462] LNetError: 130567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.132748] LNet: Removed LNI 192.168.203.106@tcp [ 2339.578810] Key type .llcrypt unregistered [ 2339.585926] Key type ._llcrypt unregistered [ 2354.747551] Key type ._llcrypt registered [ 2354.749669] Key type .llcrypt registered [ 2354.832930] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 2364.371947] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2364.971544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2364.999100] alg: No test for adler32 (adler32-zlib) [ 2365.934889] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 2366.115345] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2367.736716] Key type lgssc registered [ 2368.464722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2389.923693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2389.962406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.095988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2391.113583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2391.166765] Lustre: lustre-MDT0000: new disk, initializing [ 2391.213746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2391.227380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2394.391315] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2401.887575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.976197] Lustre: 133940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2402.016483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2402.019572] Lustre: Skipped 1 previous similar message [ 2402.067324] Lustre: lustre-MDT0001: new disk, initializing [ 2402.130228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2402.151063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2402.160826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2404.998423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2411.418638] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2414.814502] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2418.921905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2419.099371] Lustre: lustre-OST0000: new disk, initializing [ 2419.102727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2419.107727] Lustre: 135908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2419.165500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.713412] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2428.971536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2428.978915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2429.030402] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2429.068539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2431.568272] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2431.706352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2434.697899] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2434.845662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2439.136828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.140278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.144232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.150643] Lustre: Skipped 1 previous similar message [ 2439.156443] Lustre: Skipped 1 previous similar message [ 2444.258646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.264043] Lustre: Skipped 1 previous similar message [ 2449.386495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.390408] Lustre: Skipped 1 previous similar message [ 2450.399141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2450.491156] Lustre: server umount lustre-OST0000 complete [ 2453.471936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2453.476072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2453.482903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2454.498591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.802565] Lustre: server umount lustre-MDT0000 complete [ 2460.755763] LustreError: 134736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466544 with bad export cookie 4410869683786468981 [ 2460.762350] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.767753] LustreError: 134736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2460.966916] Lustre: server umount lustre-MDT0001 complete [ 2465.897447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.115068] LustreError: 138124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.132960] LustreError: 138124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2466.171512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.750642] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2471.393487] LustreError: 138125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.453597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.723737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.895555] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2481.281471] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2484.369644] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2487.630859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2487.804557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.363790] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2495.740373] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2497.974751] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2498.081187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2500.401275] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2500.534570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2506.742613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2506.852903] Lustre: lustre-OST0001: new disk, initializing [ 2506.856241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2506.861752] Lustre: 141492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2506.923495] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2510.134237] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2515.300157] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2516.509463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2516.515765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2516.556564] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2517.481155] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2519.511362] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2519.621473] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2522.210343] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0aa037e0800.ost_server_uuid 50 [ 2522.912290] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0aa037e0800.ost_server_uuid in FULL state after 0 sec [ 2526.692547] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2526.693515] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.697792] LustreError: Skipped 1 previous similar message [ 2526.704242] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2526.708168] Lustre: Skipped 3 previous similar messages [ 2532.428569] Lustre: server umount lustre-OST0001 complete [ 2534.880393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.884172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.889651] Lustre: Skipped 1 previous similar message [ 2534.892410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2534.895382] Lustre: Skipped 2 previous similar messages [ 2536.933618] LustreError: 140938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.941887] LustreError: 140938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2540.450346] Lustre: server umount lustre-OST0000 complete [ 2545.633441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2545.637765] LustreError: Skipped 1 previous similar message [ 2545.639899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.644881] Lustre: Skipped 1 previous similar message [ 2552.289351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2552.294334] Lustre: Skipped 6 previous similar messages [ 2556.383121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2556.470530] Lustre: server umount lustre-MDT0000 complete [ 2557.409658] LustreError: 138124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.423038] LustreError: 138124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2558.648515] LustreError: 138105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466642 with bad export cookie 4410869683786469856 [ 2558.652721] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.658724] LustreError: 138105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2558.869692] Lustre: server umount lustre-MDT0001 complete [ 2569.808243] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2571.404958] Key type lgssc unregistered [ 2571.556949] LNet: 144506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.562196] LNetError: 144506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2571.573649] LNet: Removed LNI 192.168.203.106@tcp [ 2572.077471] Key type .llcrypt unregistered [ 2572.079540] Key type ._llcrypt unregistered [ 2595.324307] Key type ._llcrypt registered [ 2595.326073] Key type .llcrypt registered [ 2595.385507] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 2603.754806] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2604.238199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2604.253272] alg: No test for adler32 (adler32-zlib) [ 2605.143805] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 2605.265564] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2606.863162] Key type lgssc registered [ 2607.393632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2624.785361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2624.797610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.928193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2625.946945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2625.992692] Lustre: lustre-MDT0000: new disk, initializing [ 2626.043154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.056566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2628.351648] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2633.931638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.980671] Lustre: 148635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2634.011493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2634.014287] Lustre: Skipped 1 previous similar message [ 2634.052795] Lustre: lustre-MDT0001: new disk, initializing [ 2634.105276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.123854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2634.129935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2635.949486] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2641.012874] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2643.794624] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2646.952266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.107632] Lustre: lustre-OST0000: new disk, initializing [ 2647.111202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2647.116607] Lustre: 150601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2647.164456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.127963] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2654.721808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2654.727242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2654.774326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2655.394609] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2658.818658] Lustre: Failing over lustre-OST0000 [ 2658.903749] Lustre: server umount lustre-OST0000 complete [ 2672.778944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.949385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.964463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.973949] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2673.033867] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2675.402426] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2676.063097] Lustre: 146859:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786466743/real 1786466743] req@000000009b1878e4 x1873246093864576/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786466759 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2677.121072] Lustre: Failing over lustre-MDT0000 [ 2677.227746] Lustre: server umount lustre-MDT0000 complete [ 2678.242184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.248215] Lustre: Skipped 3 previous similar messages [ 2681.311158] Lustre: 146859:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786466748/real 1786466748] req@00000000b320c27a x1873246093865856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786466764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2681.325542] Lustre: 146859:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2683.254637] LustreError: 148643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.264457] LustreError: 148643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2685.407147] Lustre: 146860:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786466753/real 1786466753] req@000000007957a3d6 x1873246093867008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786466769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2688.375672] LustreError: 152512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.390676] LustreError: 152512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2691.372340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.446754] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.454055] Lustre: 148639:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2691.549529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2691.572804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2691.597636] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2693.495837] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2693.555270] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2696.682955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2696.694625] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2696.722086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2701.794046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2701.802053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.810241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2701.815657] Lustre: Skipped 1 previous similar message [ 2704.829730] Lustre: server umount lustre-OST0000 complete [ 2712.033421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2712.034198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.039621] LustreError: Skipped 1 previous similar message [ 2712.042547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2712.057561] Lustre: Skipped 2 previous similar messages [ 2713.319254] Lustre: server umount lustre-MDT0000 complete [ 2715.574651] LustreError: 148628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786466799 with bad export cookie 13639225724211109920 [ 2715.581525] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2715.581593] LustreError: 148628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2715.798786] Lustre: server umount lustre-MDT0001 complete [ 2721.460525] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2722.976637] Key type lgssc unregistered [ 2723.138453] LNet: 154253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.143311] LNetError: 154253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2723.154575] LNet: Removed LNI 192.168.203.106@tcp [ 2723.656519] Key type .llcrypt unregistered [ 2723.659223] Key type ._llcrypt unregistered [ 2748.540301] Key type ._llcrypt registered [ 2748.542667] Key type .llcrypt registered [ 2748.594876] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2749.090508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2749.148078] alg: No test for adler32 (adler32-zlib) [ 2750.107844] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 2750.258958] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2751.871720] Key type lgssc registered [ 2752.515424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2756.512770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2756.526372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.761330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.633182] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2764.663721] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2767.063693] LustreError: 155799:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2767.068240] LustreError: 155799:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2767.071784] LustreError: 155799:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2767.208984] Lustre: server umount lustre-MDT0000 complete [ 2773.854283] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 2777.907986] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2794.449513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.549178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2794.562841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2794.601229] Lustre: lustre-MDT0000: new disk, initializing [ 2794.630753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.639663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2796.393895] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2801.938982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.996082] Lustre: 159099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2802.033668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2802.037871] Lustre: Skipped 1 previous similar message [ 2802.079135] Lustre: lustre-MDT0001: new disk, initializing [ 2802.132581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.150968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2802.156664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2804.079808] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2809.249642] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2812.246594] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2815.866817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2816.020263] Lustre: lustre-OST0000: new disk, initializing [ 2816.023185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2816.027955] Lustre: 161063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2816.078457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.344167] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2823.181228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2823.191930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2823.240305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2824.484427] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2826.919475] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2827.961450] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:48:30 (1786466910) [ 2965.983695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2965.989652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.998816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.496568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2966.505043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.508364] Lustre: Skipped 2 previous similar messages [ 2971.615858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.619457] Lustre: Skipped 2 previous similar messages [ 2976.736100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.738655] Lustre: Skipped 2 previous similar messages [ 2979.295161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2979.362916] Lustre: server umount lustre-MDT0000 complete [ 2980.827165] LustreError: 159090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467064 with bad export cookie 2081696995140733982 [ 2980.827992] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.832547] LustreError: 159090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2981.855499] LustreError: 162821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.855692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.868698] LustreError: 162821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2981.880726] Lustre: Skipped 1 previous similar message [ 2981.887666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2985.888044] LustreError: 159103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.007842] LustreError: 161781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.007913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2991.021071] Lustre: Skipped 1 previous similar message [ 2995.167173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2995.258698] Lustre: server umount lustre-MDT0001 complete [ 3003.165377] Lustre: server umount lustre-OST0000 complete [ 3007.434534] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3011.644971] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3028.890760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.036749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3029.058927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3029.110908] Lustre: lustre-MDT0000: new disk, initializing [ 3029.161382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.172440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3031.557149] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3038.065517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.142945] Lustre: 165702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3038.186952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3038.191353] Lustre: Skipped 1 previous similar message [ 3038.249487] Lustre: lustre-MDT0001: new disk, initializing [ 3038.314979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3038.339956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3038.345426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3040.627502] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3046.216759] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3049.179760] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3052.591356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.745802] Lustre: lustre-OST0000: new disk, initializing [ 3052.749168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3052.755404] Lustre: 167670:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3054.286699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3054.295024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3054.362250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3055.765092] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3060.697353] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3062.735419] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3062.830077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3064.928509] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3065.034116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3069.922310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.924983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.927283] Lustre: Skipped 1 previous similar message [ 3069.934134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3072.174880] Lustre: server umount lustre-OST0000 complete [ 3074.016661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3074.020295] LustreError: Skipped 1 previous similar message [ 3074.022282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.351133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3088.447506] Lustre: server umount lustre-MDT0000 complete [ 3090.245475] LustreError: 165694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467173 with bad export cookie 2081696995140892812 [ 3090.251206] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.253598] LustreError: 165694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3090.395818] Lustre: server umount lustre-MDT0001 complete [ 3094.080363] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:52:57 (1786467177) [ 3100.938470] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3105.140329] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3122.453561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.569476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3122.594710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3122.645790] Lustre: lustre-MDT0000: new disk, initializing [ 3122.693260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.697713] Lustre: Skipped 1 previous similar message [ 3122.706734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3124.846480] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3130.409717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.469251] Lustre: 172419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3130.615232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3132.502473] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3137.238175] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3139.747984] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3142.753585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.877449] Lustre: 174397:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3145.446915] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3148.295459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3148.347597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3149.937984] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3153.379169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.379321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.387585] Lustre: Skipped 1 previous similar message [ 3153.390230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3153.392674] Lustre: Skipped 9 previous similar messages [ 3154.200024] Lustre: server umount lustre-OST0000 complete [ 3157.800615] Lustre: server umount lustre-MDT0000 complete [ 3158.496314] LustreError: 172432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3158.503082] LustreError: 172432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3159.213379] LustreError: 172411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467242 with bad export cookie 2081696995140893694 [ 3159.216194] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3159.217907] LustreError: 172411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3159.357776] Lustre: server umount lustre-MDT0001 complete [ 3162.805689] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:54:05 (1786467245) [ 3169.718741] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3173.531268] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3188.145315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3188.233509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3188.247888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3188.250838] Lustre: Skipped 3 previous similar messages [ 3188.291340] Lustre: lustre-MDT0000: new disk, initializing [ 3188.293442] Lustre: Skipped 2 previous similar messages [ 3188.322488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.325403] Lustre: Skipped 2 previous similar messages [ 3188.332273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3188.336131] Lustre: Skipped 2 previous similar messages [ 3189.879716] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3194.544045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.591299] Lustre: 178709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3194.718199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3196.289022] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3200.520971] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3202.784846] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3205.666769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.795964] Lustre: 180687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3208.379448] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3212.737189] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3214.385856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3224.545508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3224.546467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.548844] LustreError: Skipped 3 previous similar messages [ 3224.549550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3224.554110] Lustre: Skipped 4 previous similar messages [ 3224.555730] Lustre: Skipped 1 previous similar message [ 3226.847136] Lustre: server umount lustre-OST0000 complete [ 3234.530131] Lustre: server umount lustre-MDT0000 complete [ 3234.785390] LustreError: 178721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.792897] LustreError: 178721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3236.237573] LustreError: 179509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467319 with bad export cookie 2081696995140894765 [ 3236.241986] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.244949] LustreError: 179509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3236.439206] Lustre: server umount lustre-MDT0001 complete [ 3241.297610] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3242.746534] Key type lgssc unregistered [ 3242.903834] LNet: 182542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.908255] LNetError: 182542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3243.941522] LNet: Removed LNI 192.168.203.106@tcp [ 3244.353346] Key type .llcrypt unregistered [ 3244.355321] Key type ._llcrypt unregistered [ 3255.133365] Key type ._llcrypt registered [ 3255.135060] Key type .llcrypt registered [ 3255.184408] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3255.573468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3255.580583] alg: No test for adler32 (adler32-zlib) [ 3256.465297] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 3256.584459] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3258.199195] Key type lgssc registered [ 3258.756979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3262.281839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3262.293903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.492351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.289210] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3268.787346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.941591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3270.598730] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3273.640602] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3275.740445] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3278.110062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.231616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.343057] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3283.224513] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3288.490231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3303.905326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3303.906540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.909313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3303.914352] Lustre: Skipped 1 previous similar message [ 3307.343351] Lustre: server umount lustre-OST0000 complete [ 3309.024737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.025552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.030431] Lustre: Skipped 1 previous similar message [ 3309.032728] Lustre: Skipped 1 previous similar message [ 3314.144761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.147684] Lustre: Skipped 2 previous similar messages [ 3314.713085] Lustre: server umount lustre-MDT0000 complete [ 3316.168167] LustreError: 185782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467399 with bad export cookie 13002314117954797416 [ 3316.171758] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.173071] LustreError: 185782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3316.303346] Lustre: server umount lustre-MDT0001 complete [ 3320.427519] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3321.644495] Key type lgssc unregistered [ 3321.771484] LNet: 187494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3321.774825] LNetError: 187494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3321.784470] LNet: Removed LNI 192.168.203.106@tcp [ 3322.149683] Key type .llcrypt unregistered [ 3322.151746] Key type ._llcrypt unregistered [ 3332.515202] Key type ._llcrypt registered [ 3332.518050] Key type .llcrypt registered [ 3332.572322] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3332.931903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3332.952447] alg: No test for adler32 (adler32-zlib) [ 3333.842610] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 3333.959687] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3335.559207] Key type lgssc registered [ 3336.049892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3339.571067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3339.579571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.795994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.648744] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3346.435352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.601647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3348.234141] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3351.423852] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3353.730733] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3356.273368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3356.419655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.675791] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3361.671878] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3365.675744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3370.981306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.981561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3370.982507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.989742] Lustre: Skipped 1 previous similar message [ 3372.419915] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 3372.423028] Lustre: Skipped 1 previous similar message [ 3375.515614] Lustre: server umount lustre-OST0000 complete [ 3378.300862] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa0aa10b52800.ost_server_uuid 50 [ 3378.916279] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3389.227960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.229357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.234413] Lustre: Skipped 1 previous similar message [ 3391.328614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.332025] Lustre: Skipped 2 previous similar messages [ 3395.414345] Lustre: server umount lustre-MDT0000 complete [ 3396.576333] LustreError: 188887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3396.585366] LustreError: 188887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3397.202327] LustreError: 188867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467480 with bad export cookie 9752000566609888378 [ 3397.206035] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.210156] LustreError: 188867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3397.365522] Lustre: server umount lustre-MDT0001 complete [ 3401.714639] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:58:04 (1786467484) [ 3404.832212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.081574] LustreError: 192778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.119618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3406.965739] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3410.401188] LustreError: 192779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.458457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.619338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3412.165122] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3414.792097] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3416.893948] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3419.887024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.033799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.484872] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3425.355451] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3428.359565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3428.441833] Lustre: lustre-OST0001: new disk, initializing [ 3428.444120] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3428.448131] Lustre: 195616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3429.453780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3429.458504] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3429.487269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3430.504067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3430.697324] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3434.940077] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3436.512647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.515985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.521758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3441.125446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3441.131921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3442.121072] Lustre: server umount lustre-OST0000 complete [ 3444.615661] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3444.707099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3445.622649] LustreError: 194597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.546569] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3446.634691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3450.741175] LustreError: 194598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.748247] LustreError: 194598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3460.576547] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.576596] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.579950] LustreError: Skipped 1 previous similar message [ 3460.580517] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3460.583420] Lustre: Skipped 1 previous similar message [ 3460.588012] Lustre: Skipped 4 previous similar messages [ 3461.600237] LustreError: 194597:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.605815] LustreError: 194597:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3462.667581] Lustre: server umount lustre-OST0001 complete [ 3470.113621] Lustre: server umount lustre-MDT0000 complete [ 3471.455345] LustreError: 192758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467555 with bad export cookie 9752000566609889435 [ 3471.458673] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3471.460835] LustreError: 192758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3471.583534] Lustre: server umount lustre-MDT0001 complete [ 3479.723944] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:59:22 (1786467562) [ 3481.918991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.019955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3482.037980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3482.113123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.115782] Lustre: Skipped 1 previous similar message [ 3483.512129] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3486.530405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.566231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3486.581815] Lustre: 199029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3488.120152] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3490.885296] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3492.875345] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3495.420937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3495.487104] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3497.505626] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3499.895186] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3500.559967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3502.154893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.180622] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3504.156287] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3506.620871] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3507.683995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.685856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3507.689957] Lustre: Skipped 3 previous similar messages [ 3507.694028] Lustre: Skipped 5 previous similar messages [ 3509.262926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3513.595230] Lustre: server umount lustre-OST0000 complete [ 3514.337636] LustreError: 200879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.343159] LustreError: 200879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3524.578687] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3524.579062] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.581988] LustreError: Skipped 1 previous similar message [ 3530.453014] Lustre: server umount lustre-OST0001 complete [ 3532.767950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3539.935967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3539.938142] Lustre: Skipped 11 previous similar messages [ 3546.079094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3546.147814] Lustre: server umount lustre-MDT0000 complete [ 3547.549526] LustreError: 200899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467631 with bad export cookie 9752000566609890821 [ 3547.553096] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3547.555310] LustreError: 200899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3547.688548] Lustre: server umount lustre-MDT0001 complete [ 3556.278064] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:00:39 (1786467639) [ 3561.599243] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3565.242491] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3579.731392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.844743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3579.862094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3579.900618] Lustre: lustre-MDT0000: new disk, initializing [ 3579.932407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.935190] Lustre: Skipped 3 previous similar messages [ 3579.942965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3581.597532] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3586.423045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.472940] Lustre: 207102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3586.500458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3586.502782] Lustre: Skipped 1 previous similar message [ 3586.539297] Lustre: lustre-MDT0001: new disk, initializing [ 3586.606252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3586.611189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3588.282246] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3592.846624] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3595.340287] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3598.278867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3598.396562] Lustre: lustre-OST0000: new disk, initializing [ 3598.399624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3598.404165] Lustre: 209071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3601.090402] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3603.979808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3603.990242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3604.050823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3605.617642] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3607.440742] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3607.532740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3609.343193] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3609.440416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3614.178792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3614.180232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.183255] LustreError: Skipped 1 previous similar message [ 3614.184125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3614.189855] Lustre: Skipped 4 previous similar messages [ 3614.192016] Lustre: Skipped 2 previous similar messages [ 3616.414135] Lustre: server umount lustre-OST0000 complete [ 3624.094871] Lustre: server umount lustre-MDT0000 complete [ 3624.417270] LustreError: 208024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.424924] LustreError: 208024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3625.584480] LustreError: 207096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467709 with bad export cookie 9752000566609892564 [ 3625.589351] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.589619] LustreError: 207096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3625.715640] Lustre: server umount lustre-MDT0001 complete [ 3629.132211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.922088] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3634.336732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.040472] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3638.965681] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3641.172543] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.792875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.093647] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3648.761530] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3650.250644] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3654.424264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3656.000592] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3656.088722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3659.236195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.930699] Lustre: server umount lustre-OST0000 complete [ 3665.433101] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3667.155036] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3691.033084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3691.148770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.152648] Lustre: Skipped 5 previous similar messages [ 3693.116795] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3695.537373] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3696.134554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3699.283499] Lustre: server umount lustre-OST0000 complete [ 3701.215739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.219942] Lustre: Skipped 4 previous similar messages [ 3702.689566] Lustre: server umount lustre-MDT0000 complete [ 3704.127075] LustreError: 211311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467787 with bad export cookie 9752000566609893432 [ 3704.130388] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3704.136630] LustreError: 211311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3704.259652] Lustre: server umount lustre-MDT0001 complete [ 3707.376664] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 13:03:10 (1786467790) [ 3709.972636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.132949] LustreError: 217267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3710.140682] LustreError: 217267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3711.618508] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3714.924544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.692049] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3719.413607] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3721.621520] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3724.251330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3726.656613] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3729.395459] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3730.874307] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3736.050493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3737.636916] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3737.726897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3740.179401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3740.251070] Lustre: lustre-OST0001: new disk, initializing [ 3740.253049] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3740.257136] Lustre: 220672:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3741.896298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3741.901108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3741.942217] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3742.398579] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3746.482819] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3748.169780] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3748.264671] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3749.920153] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3749.999371] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3752.418274] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.420971] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3752.421467] LustreError: Skipped 2 previous similar messages [ 3752.426728] Lustre: Skipped 9 previous similar messages [ 3756.799447] Lustre: server umount lustre-OST0001 complete [ 3758.958079] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3759.039729] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3760.536568] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3760.605018] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3784.176893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3786.186897] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3788.644863] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3789.175481] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3789.286777] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3789.292192] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3789.292320] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3789.299894] Lustre: Skipped 1 previous similar message [ 3791.932415] Lustre: server umount lustre-OST0001 complete [ 3810.127972] Lustre: server umount lustre-OST0000 complete [ 3817.691127] Lustre: server umount lustre-MDT0000 complete [ 3818.917183] LustreError: 217249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467902 with bad export cookie 9752000566609894363 [ 3818.920494] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3818.922059] LustreError: 217249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3833.311206] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3833.425840] Lustre: server umount lustre-MDT0001 complete [ 3842.431955] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 13:05:25 (1786467925) [ 3844.952711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.062749] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3845.078525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3845.168244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.171476] Lustre: Skipped 5 previous similar messages [ 3846.855574] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3850.051764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.099662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3850.113798] Lustre: 225906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3851.821355] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3854.376387] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3856.384759] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3858.941555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3859.016449] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3861.222604] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3863.864558] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3867.838927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3867.874117] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3870.031591] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3872.510980] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.085798] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3875.637590] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3875.714750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3877.265537] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3877.348052] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3879.191990] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0aa10ac2800.ost_server_uuid 50 [ 3879.696579] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0aa10ac2800.ost_server_uuid in FULL state after 0 sec [ 3880.052807] Lustre: Permanently deactivating lustre-OST0001 [ 3880.058623] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3880.939173] Lustre: Permanently reactivating lustre-OST0001 [ 3883.494085] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3883.494518] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.496861] LustreError: Skipped 4 previous similar messages [ 3883.502872] Lustre: Skipped 9 previous similar messages [ 3888.070405] Lustre: server umount lustre-OST0001 complete [ 3888.612027] LustreError: 228780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.619032] LustreError: 228780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 3895.508697] Lustre: server umount lustre-OST0000 complete [ 3911.135147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3911.198838] Lustre: server umount lustre-MDT0000 complete [ 3912.462196] LustreError: 226646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786467996 with bad export cookie 9752000566609895637 [ 3912.465673] LustreError: 226646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3912.466396] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3912.585673] Lustre: server umount lustre-MDT0001 complete [ 3920.757856] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 13:06:43 (1786468003) [ 3923.378485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.474332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3923.486597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3923.491351] Lustre: Skipped 5 previous similar messages [ 3925.244940] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3928.213335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.252621] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3928.265893] Lustre: 232588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3929.801564] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3932.689854] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3934.677268] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3937.197468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.299255] Lustre: Permanently deactivating lustre-OST0000 [ 3937.306926] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3939.071411] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3939.671342] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3942.120536] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3944.509610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.643038] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3948.210522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3949.248419] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3949.863488] Lustre: Permanently reactivating lustre-OST0000 [ 3955.710394] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3955.714884] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3955.721978] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3955.750074] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3955.761906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3963.838143] Lustre: server umount lustre-OST0001 complete [ 3967.191572] Lustre: server umount lustre-OST0000 complete [ 3974.751926] Lustre: server umount lustre-MDT0000 complete [ 3976.009046] LustreError: 232581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468059 with bad export cookie 9752000566609897268 [ 3976.013587] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3976.013702] LustreError: 232581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3976.127594] Lustre: server umount lustre-MDT0001 complete [ 3979.550768] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3980.533793] Key type lgssc unregistered [ 3980.652374] LNet: 237443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3980.656235] LNetError: 237443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3980.668259] LNet: Removed LNI 192.168.203.106@tcp [ 3980.984094] Key type .llcrypt unregistered [ 3980.985117] Key type ._llcrypt unregistered [ 3989.233802] Key type ._llcrypt registered [ 3989.234846] Key type .llcrypt registered [ 3989.268492] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3989.591239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3989.630882] alg: No test for adler32 (adler32-zlib) [ 3990.472332] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 3990.555681] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3992.135162] Key type lgssc registered [ 3992.489561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3995.918482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3995.923918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.077057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3998.617310] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4001.616865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.676739] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4001.689730] Lustre: 238875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4001.696000] Lustre: Permanently deactivating lustre-MDT0001 [ 4001.703686] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4001.856134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.218981] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4003.691690] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4003.694619] LustreError: 238917:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4003.699820] LustreError: 238917:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4003.704693] LustreError: 238917:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 4005.640508] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4007.545958] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4010.116518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4010.248134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4012.319990] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4012.326659] LustreError: 240726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.329877] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4012.333731] LustreError: 240726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4014.875347] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4017.213865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4017.276296] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4019.083515] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4021.397698] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4021.417969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4021.418157] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4021.419050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4022.998423] Lustre: Permanently reactivating lustre-MDT0001 [ 4023.004845] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4023.007256] Lustre: Skipped 2 previous similar messages [ 4031.361419] Lustre: lustre-MDT0001: Client d1f276ee-1541-4cfa-859b-692da0efaa61 (at 192.168.203.6@tcp) reconnecting [ 4034.251778] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4034.821305] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4036.662133] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4036.752823] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4038.155611] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4038.238781] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4039.054514] Lustre: Permanently deactivating lustre-MDT0001 [ 4039.061135] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4039.064320] Lustre: Skipped 2 previous similar messages [ 4046.350760] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4052.960284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4052.964186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.971218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.496381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4054.498935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.500846] Lustre: Skipped 2 previous similar messages [ 4054.506209] Lustre: Skipped 2 previous similar messages [ 4058.581691] Lustre: server umount lustre-MDT0000 complete [ 4059.616407] LustreError: 243323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.623458] LustreError: 243323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4059.940172] LustreError: 238868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468143 with bad export cookie 9472086312843247248 [ 4059.944314] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.947900] LustreError: 238868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4060.084266] Lustre: server umount lustre-MDT0001 complete [ 4067.640949] Lustre: server umount lustre-OST0000 complete [ 4071.422729] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 13:09:14 (1786468154) [ 4074.969274] Lustre: server umount lustre-OST0001 complete [ 4077.140930] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4080.486579] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4094.576524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.720188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.725155] Lustre: Skipped 2 previous similar messages [ 4094.737583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4094.779272] Lustre: lustre-MDT0000: new disk, initializing [ 4094.810764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.819332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4096.589223] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4101.462321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.508410] Lustre: 247230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4101.537109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4101.540436] Lustre: Skipped 1 previous similar message [ 4101.582448] Lustre: lustre-MDT0001: new disk, initializing [ 4101.646807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4101.653819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4103.423530] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4107.784040] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4109.982403] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4112.355353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.460899] Lustre: lustre-OST0000: new disk, initializing [ 4112.462917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4112.465505] Lustre: 249196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4112.496101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.498828] Lustre: Skipped 1 previous similar message [ 4114.526342] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4115.901764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4115.905926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4115.943887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4118.839724] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4119.689760] LustreError: 247238:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4124.412330] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4124.490651] Lustre: lustre-OST0001: new disk, initializing [ 4124.492956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4124.496567] Lustre: 250311:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4125.834273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4125.838778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4125.875722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4126.854207] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4129.791497] LustreError: 247238:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4129.802111] LustreError: 247236:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4130.973595] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4131.608630] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4139.895118] LustreError: 247236:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4141.544526] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.545039] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.547563] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4141.547569] Lustre: Skipped 1 previous similar message [ 4141.549418] Lustre: Skipped 1 previous similar message [ 4144.511112] Lustre: lustre-OST0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 4144.513506] Lustre: Skipped 1 previous similar message [ 4146.694533] Lustre: server umount lustre-OST0001 complete [ 4151.781276] LustreError: 249182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.788342] LustreError: 249182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4156.900851] LustreError: 249662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.908085] LustreError: 249662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4162.017683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.017750] LustreError: 249180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.020508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.020668] LustreError: Skipped 1 previous similar message [ 4162.021883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.021889] Lustre: Skipped 2 previous similar messages [ 4162.027111] LustreError: 249180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4165.215484] Lustre: server umount lustre-OST0000 complete [ 4167.138027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.144086] Lustre: Skipped 2 previous similar messages [ 4172.256140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.259371] Lustre: Skipped 4 previous similar messages [ 4172.958125] Lustre: server umount lustre-MDT0000 complete [ 4174.227208] LustreError: 248025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468257 with bad export cookie 9472086312843250055 [ 4174.230770] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.232036] LustreError: 248025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4180.416179] Lustre: server umount lustre-MDT0001 complete [ 4184.509702] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4185.855342] Key type lgssc unregistered [ 4186.013582] LNet: 252346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4186.019304] LNetError: 252346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4186.029388] LNet: Removed LNI 192.168.203.106@tcp [ 4186.397495] Key type .llcrypt unregistered [ 4186.399737] Key type ._llcrypt unregistered [ 4202.239325] Key type ._llcrypt registered [ 4202.240988] Key type .llcrypt registered [ 4202.285022] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4202.638395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4202.652058] alg: No test for adler32 (adler32-zlib) [ 4203.506169] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4203.607737] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4205.191109] Key type lgssc registered [ 4205.555226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4208.284204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.292214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.384673] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4209.392601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.452902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.803366] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4213.933885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.969912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4213.980622] Lustre: 254530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4214.089346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.515240] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4218.156900] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4220.109072] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4222.409635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4222.493619] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4222.568204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.665561] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4226.577756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4227.260942] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4267.489483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4267.489969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4267.494875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4267.497177] LustreError: Skipped 1 previous similar message [ 4267.500421] Lustre: Skipped 1 previous similar message [ 4272.221886] Lustre: server umount lustre-OST0000 complete [ 4273.792592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4278.351051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4278.482472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4280.598669] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4283.556727] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4287.720350] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.726190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.728369] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4287.734067] Lustre: Skipped 1 previous similar message [ 4318.688851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.691148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.691761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.692168] LustreError: Skipped 1 previous similar message [ 4318.696800] Lustre: Skipped 2 previous similar messages [ 4322.841041] Lustre: server umount lustre-OST0000 complete [ 4326.880030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4326.883049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.887496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.889235] Lustre: Skipped 1 previous similar message [ 4328.927969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.928498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.197862] Lustre: server umount lustre-MDT0000 complete [ 4331.374191] LustreError: 254522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468414 with bad export cookie 12938817232305661711 [ 4331.377764] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.378525] LustreError: 254522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4331.501532] Lustre: server umount lustre-MDT0001 complete [ 4335.394573] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4336.531253] Key type lgssc unregistered [ 4336.680765] LNet: 259946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4336.684641] LNetError: 259946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4336.695406] LNet: Removed LNI 192.168.203.106@tcp [ 4337.085278] Key type .llcrypt unregistered [ 4337.086708] Key type ._llcrypt unregistered [ 4346.186911] Key type ._llcrypt registered [ 4346.188160] Key type .llcrypt registered [ 4346.221986] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4346.529724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4346.537519] alg: No test for adler32 (adler32-zlib) [ 4347.396189] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4347.494987] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4349.079256] Key type lgssc registered [ 4349.463985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4352.146649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4352.156614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.303464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.698872] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4357.515773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.636102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4359.126846] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4361.675544] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4363.589197] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4365.858317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4365.975386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.960040] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4370.333484] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4379.302402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4389.857000] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4389.857439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.866258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4389.868147] Lustre: Skipped 1 previous similar message [ 4394.976713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4394.979063] Lustre: Skipped 1 previous similar message [ 4395.408742] Lustre: server umount lustre-OST0000 complete [ 4398.560186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4398.563464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.568448] Lustre: Skipped 1 previous similar message [ 4398.571589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.096971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.772349] Lustre: server umount lustre-MDT0000 complete [ 4404.060256] LustreError: 261319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468487 with bad export cookie 17344701183609857103 [ 4404.063048] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.064697] LustreError: 261319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4404.196188] Lustre: server umount lustre-MDT0001 complete [ 4407.895949] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4408.933763] Key type lgssc unregistered [ 4409.059513] LNet: 265665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4409.061863] LNetError: 265665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4409.074321] LNet: Removed LNI 192.168.203.106@tcp [ 4409.371916] Key type .llcrypt unregistered [ 4409.373608] Key type ._llcrypt unregistered [ 4416.276836] Key type ._llcrypt registered [ 4416.277959] Key type .llcrypt registered [ 4416.313107] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4416.601984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4416.652261] alg: No test for adler32 (adler32-zlib) [ 4417.507607] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4417.582632] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4419.159138] Key type lgssc registered [ 4419.492064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4421.799858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4421.805584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.946793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.147207] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4426.775739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.898613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4428.089500] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4430.450745] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4432.328758] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4434.480572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4434.597170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4436.435749] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4438.745029] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4447.909158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4453.345615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.347087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4453.350060] Lustre: Skipped 1 previous similar message [ 4454.673740] Lustre: server umount lustre-OST0000 complete [ 4457.951911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4457.954038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.958555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.960577] Lustre: Skipped 1 previous similar message [ 4461.972611] Lustre: server umount lustre-MDT0000 complete [ 4463.155076] LustreError: 266844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468546 with bad export cookie 18068364278317646158 [ 4463.158274] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4463.160368] LustreError: 266844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4463.278463] Lustre: server umount lustre-MDT0001 complete [ 4466.937045] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4467.947086] Key type lgssc unregistered [ 4468.074410] LNet: 270558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4468.077456] LNetError: 270558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4468.088337] LNet: Removed LNI 192.168.203.106@tcp [ 4468.405160] Key type .llcrypt unregistered [ 4468.406541] Key type ._llcrypt unregistered [ 4477.471278] Key type ._llcrypt registered [ 4477.472698] Key type .llcrypt registered [ 4477.515370] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4477.836543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4477.885099] alg: No test for adler32 (adler32-zlib) [ 4478.738519] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4478.824256] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4480.407132] Key type lgssc registered [ 4480.737374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4483.064125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4483.070492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.202857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.507821] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4488.201363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4488.305703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4489.580655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4491.823873] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4493.515374] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4495.658221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4495.792565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.601276] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4499.803760] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4509.098979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4518.367789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4518.369807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.373369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4519.395299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.395520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4519.400825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4519.402165] Lustre: Skipped 1 previous similar message [ 4523.027613] Lustre: server umount lustre-OST0000 complete [ 4524.513654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4524.513780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.521641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4524.523785] Lustre: Skipped 1 previous similar message [ 4529.632720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4529.636135] Lustre: Skipped 1 previous similar message [ 4530.396777] Lustre: server umount lustre-MDT0000 complete [ 4531.645031] LustreError: 271933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468615 with bad export cookie 10642574082253445474 [ 4531.648479] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.650144] LustreError: 271933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4531.770687] Lustre: server umount lustre-MDT0001 complete [ 4535.547074] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4536.577451] Key type lgssc unregistered [ 4536.707417] LNet: 276275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4536.711252] LNetError: 276275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4536.721291] LNet: Removed LNI 192.168.203.106@tcp [ 4537.031251] Key type .llcrypt unregistered [ 4537.032430] Key type ._llcrypt unregistered [ 4543.926507] Key type ._llcrypt registered [ 4543.927457] Key type .llcrypt registered [ 4543.960518] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4544.322027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4544.337806] alg: No test for adler32 (adler32-zlib) [ 4545.178477] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4545.251907] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4546.831128] Key type lgssc registered [ 4547.193250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4549.651564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4549.657984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.823280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4552.118300] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4554.986315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.120734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4556.423326] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4558.589163] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4560.209519] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4562.340981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4562.460511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.300650] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4566.606747] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4575.784277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4580.834698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.836164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.836373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.840153] Lustre: Skipped 1 previous similar message [ 4582.608168] Lustre: server umount lustre-OST0000 complete [ 4585.953899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.954317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4585.959039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4585.964285] Lustre: Skipped 2 previous similar messages [ 4589.970706] Lustre: server umount lustre-MDT0000 complete [ 4591.073421] LustreError: 277495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.078637] LustreError: 277495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4591.229457] LustreError: 277454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468674 with bad export cookie 4683598979646197841 [ 4591.232924] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4591.235294] LustreError: 277454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4591.365703] Lustre: server umount lustre-MDT0001 complete [ 4595.052029] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4596.161806] Key type lgssc unregistered [ 4596.288371] LNet: 281208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4596.290894] LNetError: 281208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4596.300402] LNet: Removed LNI 192.168.203.106@tcp [ 4596.640887] Key type .llcrypt unregistered [ 4596.642064] Key type ._llcrypt unregistered [ 4600.355570] Key type ._llcrypt registered [ 4600.357302] Key type .llcrypt registered [ 4600.401625] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4605.987322] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4612.135181] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4612.595037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4612.611247] alg: No test for adler32 (adler32-zlib) [ 4613.472497] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4613.564478] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4615.151097] Key type lgssc registered [ 4615.525796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.078371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.085019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.155444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4627.164696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4627.198553] Lustre: lustre-MDT0000: new disk, initializing [ 4627.216719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.221581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4628.432327] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4632.443500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.480383] Lustre: 284733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4632.500392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4632.502318] Lustre: Skipped 1 previous similar message [ 4632.538026] Lustre: lustre-MDT0001: new disk, initializing [ 4632.571053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4632.582815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4632.586434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4633.764744] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4637.526718] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4639.181967] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4641.261767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4641.356417] Lustre: lustre-OST0000: new disk, initializing [ 4641.358667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4641.361844] Lustre: 286699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4641.393441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.050924] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4646.759985] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4648.205336] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4648.952873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4648.956127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4648.987906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4649.304869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4650.726352] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4650.794266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4654.050168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4654.050355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.052716] LustreError: Skipped 1 previous similar message [ 4654.056989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4654.059620] Lustre: Skipped 1 previous similar message [ 4657.555570] Lustre: server umount lustre-OST0000 complete [ 4659.169526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.170547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4659.173422] Lustre: Skipped 2 previous similar messages [ 4664.288543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4664.291662] Lustre: Skipped 2 previous similar messages [ 4665.061277] Lustre: server umount lustre-MDT0000 complete [ 4666.304214] LustreError: 284724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468749 with bad export cookie 15540552133428993554 [ 4666.307736] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4666.309327] LustreError: 284724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4666.441214] Lustre: server umount lustre-MDT0001 complete [ 4669.223845] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:19:12 (1786468752) [ 4671.394510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.582049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.963787] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4675.899611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.016931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4677.416586] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4679.821302] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4681.644532] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4684.012235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4684.129473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.067610] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4688.482695] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4706.786290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4706.787433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.789300] LustreError: Skipped 1 previous similar message [ 4706.793367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4706.796846] Lustre: Skipped 1 previous similar message [ 4707.858319] Lustre: server umount lustre-OST0000 complete [ 4711.904346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4711.905468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.905989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4711.913198] Lustre: Skipped 2 previous similar messages [ 4715.346155] Lustre: server umount lustre-MDT0000 complete [ 4716.471981] LustreError: 289141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468800 with bad export cookie 15540552133428994422 [ 4716.474524] LustreError: 289852:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4716.475293] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4716.477559] LustreError: 289141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4716.578393] Lustre: server umount lustre-MDT0001 complete [ 4720.141495] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4721.218561] Key type lgssc unregistered [ 4721.353514] LNet: 292670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4721.356617] LNetError: 292670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4721.366307] LNet: Removed LNI 192.168.203.106@tcp [ 4721.702783] Key type .llcrypt unregistered [ 4721.704419] Key type ._llcrypt unregistered [ 4729.951668] Key type ._llcrypt registered [ 4729.952740] Key type .llcrypt registered [ 4729.987148] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4735.973208] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4736.406985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4736.429451] alg: No test for adler32 (adler32-zlib) [ 4737.290452] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4737.385927] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4738.967227] Key type lgssc registered [ 4739.351752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4754.315710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4754.321535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.406798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4755.439068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4755.479805] Lustre: lustre-MDT0000: new disk, initializing [ 4755.507164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.514734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4757.019146] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4761.244767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.287503] Lustre: 296465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4761.318316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4761.320512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4761.322671] Lustre: Skipped 1 previous similar message [ 4761.358917] Lustre: lustre-MDT0001: new disk, initializing [ 4761.391269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4761.403192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4761.406816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4762.757628] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4766.405637] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4768.178252] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4770.390086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4770.492498] Lustre: lustre-OST2710: new disk, initializing [ 4770.495336] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4770.499552] Lustre: 298433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4770.528506] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4772.448313] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4774.516841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4774.520024] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4774.551323] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4775.957645] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4778.111864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4778.168730] Lustre: lustre-OST03e8: new disk, initializing [ 4778.170339] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4778.172692] Lustre: 299494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4778.198900] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4780.157295] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4782.596202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4782.600359] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4782.626195] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4782.634387] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4785.267857] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4785.344470] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4786.710922] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4786.788733] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4788.127816] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4788.194385] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4789.580260] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4789.651453] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4792.288603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4792.291839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.297215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4792.803743] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4792.809095] Lustre: Skipped 1 previous similar message [ 4796.833756] Lustre: server umount lustre-MDT0000 complete [ 4798.067093] LustreError: 296459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468881 with bad export cookie 15936628237460799116 [ 4798.069826] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.070832] LustreError: 296459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4798.185452] Lustre: server umount lustre-MDT0001 complete [ 4810.241457] Lustre: server umount lustre-OST2710 complete [ 4821.720673] Lustre: server umount lustre-OST03e8 complete [ 4827.311910] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 4830.475236] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4842.368882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.455252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4842.469554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4842.505400] Lustre: lustre-MDT0000: new disk, initializing [ 4842.529814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.537620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4843.934929] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4848.163608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.208658] Lustre: 304733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4848.308716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4849.689968] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4853.326519] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4855.071383] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4857.222705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4857.318117] Lustre: lustre-OST0000: new disk, initializing [ 4857.319542] Lustre: Skipped 1 previous similar message [ 4857.320864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4857.323385] Lustre: Skipped 2 previous similar messages [ 4857.325467] Lustre: 306699:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4857.353104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4857.355147] Lustre: Skipped 1 previous similar message [ 4859.137424] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4862.711048] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4864.055925] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4865.011520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4865.014067] Lustre: Skipped 1 previous similar message [ 4865.015917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4865.039100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4865.141788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4866.424250] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4866.494075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4870.112452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4870.112878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4870.115143] LustreError: Skipped 1 previous similar message [ 4870.118629] Lustre: Skipped 1 previous similar message [ 4870.120558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4870.124215] Lustre: Skipped 4 previous similar messages [ 4873.167447] Lustre: server umount lustre-OST0000 complete [ 4875.232372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.233124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4875.236741] Lustre: Skipped 1 previous similar message [ 4880.352203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4880.355047] Lustre: Skipped 2 previous similar messages [ 4880.418798] Lustre: server umount lustre-MDT0000 complete [ 4881.649341] LustreError: 304724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786468965 with bad export cookie 15936628237460800796 [ 4881.652336] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4881.654216] LustreError: 304724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4881.784651] Lustre: server umount lustre-MDT0001 complete [ 4884.551884] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:22:47 (1786468967) [ 4885.032643] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4885.540305] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:22:48 (1786468968) [ 4886.553065] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4895.061402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.134508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4895.141973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4895.197541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.414182] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4899.222390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4899.269104] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4899.280052] Lustre: 310558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4900.731098] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4916.578687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4916.647092] Lustre: Denying initial registration attempt from nid 192.168.203.106@tcp, specified as failover [ 4916.650418] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4916.654035] LustreError: 312009:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4916.657446] LustreError: 312009:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4916.659362] LustreError: 312009:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4916.680802] Lustre: server umount lustre-OST0000 complete [ 4916.683231] LustreError: 312009:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4919.776035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4919.778183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.782401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4925.007893] Lustre: server umount lustre-MDT0000 complete [ 4926.244023] LustreError: 310548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469009 with bad export cookie 15936628237460801664 [ 4926.246481] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4926.247784] LustreError: 310548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4926.366449] Lustre: server umount lustre-MDT0001 complete [ 4929.826663] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4930.786586] Key type lgssc unregistered [ 4930.919282] LNet: 313132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4930.921490] LNetError: 313132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4930.932305] LNet: Removed LNI 192.168.203.106@tcp [ 4931.206215] Key type .llcrypt unregistered [ 4931.207182] Key type ._llcrypt unregistered [ 4934.643244] Key type ._llcrypt registered [ 4934.644178] Key type .llcrypt registered [ 4934.676188] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4934.955414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4934.970944] alg: No test for adler32 (adler32-zlib) [ 4935.809933] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 4935.892824] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 4937.471123] Key type lgssc registered [ 4937.821363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4961.094547] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 4963.767193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4963.773661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4964.857409] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4964.864127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4964.913740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.211896] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4968.903502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.935504] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4968.945710] Lustre: 315870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4969.035730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.358789] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4972.657103] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4974.282467] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4989.346854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4989.422629] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4989.490326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.411960] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4993.795912] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4994.490617] Lustre: server umount lustre-OST0000 complete [ 4999.647892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4999.652111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4999.659714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5000.672638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5000.672872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.815078] Lustre: server umount lustre-MDT0000 complete [ 5003.041979] LustreError: 315863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469086 with bad export cookie 16596672684549577133 [ 5003.045076] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5003.045826] LustreError: 315863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5003.160150] Lustre: server umount lustre-MDT0001 complete [ 5006.899503] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5007.945551] Key type lgssc unregistered [ 5008.084299] LNet: 319438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5008.086632] LNetError: 319438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5008.093334] LNet: Removed LNI 192.168.203.106@tcp [ 5008.403259] Key type .llcrypt unregistered [ 5008.404625] Key type ._llcrypt unregistered [ 5017.865060] Key type ._llcrypt registered [ 5017.865930] Key type .llcrypt registered [ 5017.897177] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5018.230215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5018.263071] alg: No test for adler32 (adler32-zlib) [ 5019.116708] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5019.203453] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5020.783171] Key type lgssc registered [ 5021.206644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5025.812632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5025.828244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.064722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5030.606647] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5036.516834] LustreError: 320832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.530480] LustreError: 320832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5036.821150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.033348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5040.209617] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5046.129692] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5050.512955] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5057.410321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5057.894622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5062.565715] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5067.688767] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5077.987668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5077.992593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.002141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.501482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.506881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.517991] Lustre: Skipped 1 previous similar message [ 5078.522033] Lustre: Skipped 2 previous similar messages [ 5079.474226] Lustre: server umount lustre-MDT0000 complete [ 5081.634981] LustreError: 320814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469165 with bad export cookie 5761723583916684997 [ 5081.643256] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5081.643674] LustreError: 320814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5083.514262] LustreError: 322651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.530259] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 5083.616966] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.901138] Lustre: server umount lustre-MDT0001 complete [ 5088.069882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5092.635755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.108416] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4ff5c112995852c5 [ 5107.115794] Lustre: MGC192.168.203.106@tcp: Connection restored to 0@lo (at 0@lo) [ 5107.247587] LustreError: 322663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.256825] LustreError: 322663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5107.288880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5107.318080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5109.325826] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5112.288892] LustreError: 322664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.433366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.625987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5115.549604] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5119.191188] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5121.061078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 5122.024334] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5123.801808] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5124.554196] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5127.140100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.142330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.150676] Lustre: Skipped 1 previous similar message [ 5127.158090] Lustre: Skipped 2 previous similar messages [ 5132.257390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.262190] Lustre: Skipped 1 previous similar message [ 5132.921859] Lustre: server umount lustre-OST0000 complete [ 5137.377083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5137.390138] Lustre: Skipped 1 previous similar message [ 5141.233231] Lustre: server umount lustre-MDT0000 complete [ 5142.496980] LustreError: 324426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.509157] LustreError: 324426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5143.225760] LustreError: 322676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469226 with bad export cookie 5761723583916692165 [ 5143.229649] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5143.234494] LustreError: 322676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5143.410189] Lustre: server umount lustre-MDT0001 complete [ 5148.566558] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5149.830114] Key type lgssc unregistered [ 5149.994781] LNet: 327094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5149.999167] LNetError: 327094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5150.009604] LNet: Removed LNI 192.168.203.106@tcp [ 5150.434352] Key type .llcrypt unregistered [ 5150.435926] Key type ._llcrypt unregistered [ 5162.479308] Key type ._llcrypt registered [ 5162.486074] Key type .llcrypt registered [ 5162.551213] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5163.007299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5163.035057] alg: No test for adler32 (adler32-zlib) [ 5163.921052] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5164.033051] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5165.631227] Key type lgssc registered [ 5166.145252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5169.521783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5169.529249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.717968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5172.410983] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5176.533496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5176.708246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5178.746417] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5182.134198] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5184.764367] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5187.040145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5187.043768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.050469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5188.065734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5188.067154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5192.313418] Lustre: server umount lustre-MDT0000 complete [ 5193.185103] LustreError: 329670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.197415] LustreError: 329670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5194.072229] LustreError: 328469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469277 with bad export cookie 16954189008414706861 [ 5194.076129] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.077804] LustreError: 328469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5194.201943] Lustre: server umount lustre-MDT0001 complete [ 5200.496183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.627136] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5200.660091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5200.758144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.617829] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5206.489848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.544893] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5206.559566] Lustre: 331074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5206.682541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5208.634275] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5212.162297] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5214.768577] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5220.328527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.441340] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5220.447337] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5220.451495] LustreError: 332957:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5220.455674] LustreError: 332957:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5220.458359] LustreError: 332957:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5220.492555] Lustre: server umount lustre-OST0000 complete [ 5220.495393] LustreError: 332957:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5226.213803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5226.315685] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5226.434218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.427212] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5233.135921] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5236.302580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5237.145401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.199931] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5237.247523] Lustre: lustre-OST0001: new disk, initializing [ 5237.249813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5237.253967] Lustre: 334497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5237.306564] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5240.428248] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5241.871627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5241.881987] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5241.949934] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5243.876972] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5246.947962] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.948195] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5246.949067] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5246.949071] Lustre: Skipped 1 previous similar message [ 5246.952843] Lustre: Skipped 1 previous similar message [ 5246.955742] LustreError: Skipped 1 previous similar message [ 5251.469581] Lustre: server umount lustre-OST0001 complete [ 5252.072330] LustreError: 333423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5252.086454] LustreError: 333423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5256.673409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5256.673429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.678075] LustreError: Skipped 1 previous similar message [ 5256.678649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5256.684710] Lustre: Skipped 1 previous similar message [ 5256.687016] Lustre: Skipped 1 previous similar message [ 5257.191402] LustreError: 333423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5257.199787] LustreError: 333423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5259.551209] Lustre: server umount lustre-OST0000 complete [ 5262.305447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.314475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5262.318222] Lustre: Skipped 4 previous similar messages [ 5267.593860] Lustre: server umount lustre-MDT0000 complete [ 5269.322634] LustreError: 331065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469352 with bad export cookie 16954189008414707400 [ 5269.326751] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.333252] LustreError: 331065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5269.489501] Lustre: server umount lustre-MDT0001 complete [ 5274.887653] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5276.405611] Key type lgssc unregistered [ 5276.558138] LNet: 336453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5276.562755] LNetError: 336453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5276.573699] LNet: Removed LNI 192.168.203.106@tcp [ 5277.000943] Key type .llcrypt unregistered [ 5277.003044] Key type ._llcrypt unregistered [ 5317.347475] Key type ._llcrypt registered [ 5317.348901] Key type .llcrypt registered [ 5317.405845] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 5325.873301] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5326.288311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5326.323096] alg: No test for adler32 (adler32-zlib) [ 5327.277978] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5327.416456] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5329.023170] Key type lgssc registered [ 5329.706735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5347.699484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5347.710185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.833122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5348.854250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5348.899513] Lustre: lustre-MDT0000: new disk, initializing [ 5348.935347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.943880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5351.108858] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5357.185583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.269324] Lustre: 341375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5357.317789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5357.321544] Lustre: Skipped 1 previous similar message [ 5357.372760] Lustre: lustre-MDT0001: new disk, initializing [ 5357.427761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5357.449563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5357.458875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5359.503032] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5364.502698] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5367.138760] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5370.502111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5370.655171] Lustre: lustre-OST0000: new disk, initializing [ 5370.658838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5370.660788] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5370.665359] Lustre: 343343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5370.716505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5373.654090] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5376.012152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5376.021788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5376.043960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5379.005069] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5381.040458] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5381.160303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5383.115649] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5383.236743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5385.703257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5385.703733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5385.721209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5385.727107] Lustre: Skipped 1 previous similar message [ 5386.338764] Lustre: server umount lustre-OST0000 complete [ 5390.819679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5390.822219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5390.830749] Lustre: Skipped 2 previous similar messages [ 5390.837060] Lustre: Skipped 1 previous similar message [ 5394.484822] Lustre: server umount lustre-MDT0000 complete [ 5396.585643] LustreError: 343339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469480 with bad export cookie 15312876657497732829 [ 5396.589181] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5396.597888] LustreError: 343339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5396.784482] Lustre: server umount lustre-MDT0001 complete [ 5401.289280] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:31:24 (1786469484) [ 5405.034742] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:31:27 (1786469487) [ 5410.128139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.371833] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5410.384810] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5410.425264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.112488] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5415.908808] LustreError: 346090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5418.514038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.743622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5421.521701] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5426.137266] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5429.923574] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5434.310372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5434.507995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.423901] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5442.970190] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5449.490075] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5450.554908] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5451.558519] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5452.549751] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5453.711113] Lustre: Failing over lustre-MDT0000 [ 5454.308147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.310535] LustreError: 346086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.320660] Lustre: Skipped 1 previous similar message [ 5454.333454] LustreError: 346086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5455.873417] Lustre: server umount lustre-MDT0000 complete [ 5459.424679] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.431183] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5464.544727] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5464.555717] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5469.471154] Lustre: 339601:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786469537/real 1786469537] req@000000006054ce7c x1873248949165696/t0(0) o400->MGC192.168.203.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786469553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5469.481031] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.213573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.785393] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5474.799817] LustreError: 346089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5479.904565] LustreError: 339599:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000586a9a9 x1873248949170816/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5480.032965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5480.062991] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5481.335721] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5482.204404] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5485.031698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5485.052810] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5485.077597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:65) [ 5487.713323] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5488.519841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5490.803817] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5491.772788] Lustre: Failing over lustre-MDT0000 [ 5491.902504] Lustre: server umount lustre-MDT0000 complete [ 5495.265378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.266153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5495.268753] LustreError: 346085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5495.268765] LustreError: 346085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5495.273047] Lustre: Skipped 3 previous similar messages [ 5506.079763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.148691] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.293501] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5506.933696] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5508.261348] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5511.658880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5511.662272] Lustre: Skipped 2 previous similar messages [ 5511.676409] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5511.700131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:97) [ 5514.230897] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5514.980583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5521.890300] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.891697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5521.896348] Lustre: Skipped 1 previous similar message [ 5521.902228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5524.084591] Lustre: server umount lustre-MDT0000 complete [ 5525.763362] LustreError: 346071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469609 with bad export cookie 15312876657497735538 [ 5525.767540] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.773178] LustreError: 346071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5525.907074] Lustre: server umount lustre-MDT0001 complete [ 5537.838968] Lustre: server umount lustre-OST0000 complete [ 5544.941775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5545.143653] LustreError: 352423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.156033] LustreError: 352423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5545.178975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.182988] Lustre: Skipped 1 previous similar message [ 5547.144990] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5551.485045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.732818] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5557.374940] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5560.247180] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5563.788646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5566.816845] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5570.150292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:129) [ 5570.161221] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5582.818382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.823209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5582.823579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.823583] Lustre: Skipped 2 previous similar messages [ 5582.827558] Lustre: Skipped 2 previous similar messages [ 5588.592194] Lustre: server umount lustre-OST0000 complete [ 5592.543943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5592.548340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5592.551542] Lustre: Skipped 3 previous similar messages [ 5596.639879] Lustre: server umount lustre-MDT0000 complete [ 5598.380700] LustreError: 352405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469681 with bad export cookie 15312876657497736287 [ 5598.383500] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5598.390158] LustreError: 352405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5598.537373] Lustre: server umount lustre-MDT0001 complete [ 5603.162139] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5604.590172] Key type lgssc unregistered [ 5604.752369] LNet: 355975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5604.757294] LNetError: 355975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5604.766663] LNet: Removed LNI 192.168.203.106@tcp [ 5605.259595] Key type .llcrypt unregistered [ 5605.261216] Key type ._llcrypt unregistered [ 5617.284312] Key type ._llcrypt registered [ 5617.286188] Key type .llcrypt registered [ 5617.334797] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5617.713465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5617.767202] alg: No test for adler32 (adler32-zlib) [ 5618.636519] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5618.743810] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5620.335120] Key type lgssc registered [ 5620.791553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5623.833818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5623.841797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.015218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5626.576746] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5630.062119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.210802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5631.829919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5634.664678] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.901918] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5639.830246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.967729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.514777] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5645.941967] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5650.219605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:35 to 0x280000400:161) [ 5665.761804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.766124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5665.770079] Lustre: Skipped 2 previous similar messages [ 5665.779875] Lustre: Skipped 2 previous similar messages [ 5667.581195] Lustre: server umount lustre-MDT0000 complete [ 5669.369453] LustreError: 357449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469752 with bad export cookie 10790974601468639850 [ 5669.374669] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.380666] LustreError: 357449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5669.511921] Lustre: server umount lustre-MDT0001 complete [ 5680.630747] Lustre: server umount lustre-OST0000 complete [ 5686.970638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.182624] LustreError: 361194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.196705] LustreError: 361194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5687.223178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.174383] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5692.384773] LustreError: 361195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.124840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.289420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.183645] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5698.129612] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5700.694234] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5703.941573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5704.094443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.623717] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5709.719230] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5712.298441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 5722.593845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.599948] Lustre: Skipped 2 previous similar messages [ 5722.600806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.605339] Lustre: Skipped 2 previous similar messages [ 5727.712463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.717691] Lustre: Skipped 2 previous similar messages [ 5728.746776] Lustre: server umount lustre-MDT0000 complete [ 5730.288544] LustreError: 361176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469813 with bad export cookie 10790974601468640956 [ 5730.292540] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5730.293903] LustreError: 361176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5730.422455] Lustre: server umount lustre-MDT0001 complete [ 5742.450531] Lustre: server umount lustre-OST0000 complete [ 5749.579841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.789167] LustreError: 365102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.830199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.650623] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5754.850545] LustreError: 365103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.179856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5756.960602] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5760.163747] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5762.716521] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5765.882536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5768.444810] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5771.429311] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5775.270548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:225) [ 5779.876671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5779.876803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5783.957142] Lustre: server umount lustre-OST0000 complete [ 5786.080273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5786.084035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5786.089647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5786.092701] Lustre: Skipped 1 previous similar message [ 5790.177477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.579362] Lustre: server umount lustre-MDT0000 complete [ 5793.025292] LustreError: 366942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786469876 with bad export cookie 10790974601468641964 [ 5793.028291] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5793.029916] LustreError: 366942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5793.155633] Lustre: server umount lustre-MDT0001 complete [ 5797.401765] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5798.666512] Key type lgssc unregistered [ 5798.819391] LNet: 368632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5798.822630] LNetError: 368632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5798.831586] LNet: Removed LNI 192.168.203.106@tcp [ 5799.181062] Key type .llcrypt unregistered [ 5799.182825] Key type ._llcrypt unregistered [ 5810.296262] Key type ._llcrypt registered [ 5810.297486] Key type .llcrypt registered [ 5810.338215] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5810.649100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5810.660239] alg: No test for adler32 (adler32-zlib) [ 5811.529347] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5811.639370] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5813.231270] Key type lgssc registered [ 5813.676712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5816.685745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5816.693743] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.697227] LustreError: 370048:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5816.706275] LustreError: 370048:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5816.709819] LustreError: 370048:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5816.713227] LustreError: 370048:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5816.717662] LustreError: 370048:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5819.853035] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.857176] LustreError: 370445:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5819.867850] LustreError: 370445:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5819.871433] LustreError: 370445:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5819.876396] LustreError: 370445:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5819.883191] LustreError: 370445:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5827.421863] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5828.635913] Key type lgssc unregistered [ 5828.770089] LNet: 371449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5828.774614] LNetError: 371449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5828.783536] LNet: Removed LNI 192.168.203.106@tcp [ 5829.150482] Key type .llcrypt unregistered [ 5829.152368] Key type ._llcrypt unregistered [ 5836.113721] Key type ._llcrypt registered [ 5836.115217] Key type .llcrypt registered [ 5836.154564] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 5843.332771] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5843.663159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5843.671626] alg: No test for adler32 (adler32-zlib) [ 5844.548220] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 5844.649022] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5846.239161] Key type lgssc registered [ 5846.675759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5860.969590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5860.982519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.113909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5862.131674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5862.174141] Lustre: lustre-MDT0000: new disk, initializing [ 5862.237425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.246066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5864.064416] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5869.092017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.150567] Lustre: 374587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5869.187913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5869.190353] Lustre: Skipped 1 previous similar message [ 5869.235047] Lustre: lustre-MDT0001: new disk, initializing [ 5869.277328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5869.293435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5869.299438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5870.941398] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5875.677126] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5878.189434] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5881.164790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5881.281170] Lustre: lustre-OST0000: new disk, initializing [ 5881.283800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5881.287229] Lustre: 376555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5881.326181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.819773] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5888.205717] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5890.057340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5890.062733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5890.073512] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5890.102646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5890.171958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5892.019753] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5892.129244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5895.137186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5895.142804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5895.150795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5895.154779] Lustre: Skipped 1 previous similar message [ 5899.103051] Lustre: server umount lustre-OST0000 complete [ 5905.378304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.379062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5905.379669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5905.383891] Lustre: Skipped 2 previous similar messages [ 5905.392556] LustreError: Skipped 1 previous similar message [ 5910.496177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5910.498632] Lustre: Skipped 2 previous similar messages [ 5914.593173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5914.596488] Lustre: Skipped 1 previous similar message [ 5915.103187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5915.180062] Lustre: server umount lustre-MDT0000 complete [ 5916.718295] LustreError: 375380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470000 with bad export cookie 14132242774921618457 [ 5916.720071] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5916.724265] LustreError: 375380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5916.861515] Lustre: server umount lustre-MDT0001 complete [ 5920.212962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5920.885558] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:40:03 (1786470003) [ 5923.506193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.733819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5925.405762] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5928.741975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.909461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5930.599605] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5933.532894] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5935.752959] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5938.465237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.607114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5940.861933] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5944.026554] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.911426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5946.992069] Lustre: lustre-OST0001: new disk, initializing [ 5946.994680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5946.997753] Lustre: 381845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5949.656986] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5952.011619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5952.016210] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5952.050755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5954.111859] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5957.088728] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5957.090778] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.094587] LustreError: Skipped 1 previous similar message [ 5957.102900] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5957.106863] Lustre: Skipped 1 previous similar message [ 5958.295251] Lustre: server umount lustre-OST0001 complete [ 5959.142807] LustreError: 382375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.151861] LustreError: 382375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5962.212462] LustreError: 380831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5962.220689] LustreError: 380831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5967.331185] LustreError: 380832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.338312] LustreError: 380832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5972.449280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.451386] LustreError: 382375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.452815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5972.454981] Lustre: Skipped 2 previous similar messages [ 5972.465203] LustreError: 382375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5977.571049] LustreError: 380831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.578934] LustreError: 380831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5985.247901] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5985.301449] Lustre: server umount lustre-OST0000 complete [ 5987.808685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.812852] Lustre: Skipped 1 previous similar message [ 5992.790537] Lustre: server umount lustre-MDT0000 complete [ 5992.928057] LustreError: 379015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.936665] LustreError: 379015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5994.075424] LustreError: 381844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470077 with bad export cookie 14132242774921619325 [ 5994.079065] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5994.080034] LustreError: 381844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6000.238906] Lustre: server umount lustre-MDT0001 complete [ 6004.115207] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6005.284046] Key type lgssc unregistered [ 6005.418499] LNet: 383836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6005.421795] LNetError: 383836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6005.430556] LNet: Removed LNI 192.168.203.106@tcp [ 6005.751213] Key type .llcrypt unregistered [ 6005.752904] Key type ._llcrypt unregistered [ 6024.803304] Key type ._llcrypt registered [ 6024.804677] Key type .llcrypt registered [ 6024.839358] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6025.222953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6025.240969] alg: No test for adler32 (adler32-zlib) [ 6026.091931] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6026.177728] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6027.759151] Key type lgssc registered [ 6028.172045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6030.943814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6030.954051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.050100] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6032.058768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6032.113835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6033.492444] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6036.392279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.432897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6036.445079] Lustre: 386609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6036.538710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6038.022639] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6040.779535] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6043.173158] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6045.152286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.152922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6045.158707] Lustre: Skipped 1 previous similar message [ 6050.127970] Lustre: server umount lustre-MDT0000 complete [ 6051.410233] LustreError: 386601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470135 with bad export cookie 2788711895130713005 [ 6051.413207] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.415222] LustreError: 386601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6051.532552] Lustre: server umount lustre-MDT0001 complete [ 6054.734631] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:42:17 (1786470137) [ 6057.054633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.216058] LustreError: 389120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.222592] LustreError: 389120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6057.235783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.790188] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6061.997571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.141149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6063.756855] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6066.588556] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6068.856082] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6071.466577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6071.563508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6071.639296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.790299] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6076.417681] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6085.561490] Lustre: Permanently deactivating lustre-OST0000 [ 6085.567567] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6087.097410] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6087.661185] LustreError: 391960:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6087.665747] LustreError: 391960:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6087.668699] LustreError: 391960:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6088.436332] LustreError: 392008:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6088.439421] LustreError: 392008:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6095.245683] Lustre: server umount lustre-OST0000 complete [ 6096.182690] LustreError: 392257:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 6096.186257] LustreError: 392257:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6096.863879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.864245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6096.869144] Lustre: Skipped 1 previous similar message [ 6096.872949] Lustre: Skipped 2 previous similar messages [ 6101.984025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6101.986613] Lustre: Skipped 1 previous similar message [ 6107.104670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6107.106597] Lustre: Skipped 1 previous similar message [ 6111.199092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6111.258204] Lustre: server umount lustre-MDT0000 complete [ 6112.224057] LustreError: 389121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.228771] LustreError: 389121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6112.408168] LustreError: 389850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470196 with bad export cookie 2788711895130713572 [ 6112.410466] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6112.412950] LustreError: 389850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6123.039964] Lustre: server umount lustre-MDT0001 complete [ 6126.363618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.976729] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6129.120104] LustreError: 393414:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6131.742654] Lustre: 393754:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.106@tcp [ 6132.538101] Lustre: server umount MGS complete [ 6136.593401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.777112] LustreError: 394477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.785838] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6136.788104] Lustre: Skipped 1 previous similar message [ 6136.813928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.305961] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6141.697092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.831731] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6143.495914] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6146.136652] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6148.109210] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6148.752047] Lustre: Permanently reactivating lustre-OST0000 [ 6148.757135] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6148.759510] Lustre: Skipped 2 previous similar messages [ 6171.104317] LustreError: 396159:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6175.086749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.216398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.218848] Lustre: Skipped 1 previous similar message [ 6177.606506] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6180.254689] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6190.560640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6190.562302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.563608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6190.571103] Lustre: Skipped 1 previous similar message [ 6193.163973] Lustre: server umount lustre-OST0000 complete [ 6195.680302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.601632] Lustre: server umount lustre-MDT0000 complete [ 6200.800179] LustreError: 395204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6200.806757] LustreError: 395204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6201.896171] LustreError: 395211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470285 with bad export cookie 2788711895130714608 [ 6201.899279] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6201.901470] LustreError: 395211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6202.021663] Lustre: server umount lustre-MDT0001 complete [ 6205.832716] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6206.897327] Key type lgssc unregistered [ 6207.021405] LNet: 399031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6207.024481] LNetError: 399031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6207.034290] LNet: Removed LNI 192.168.203.106@tcp [ 6207.325748] Key type .llcrypt unregistered [ 6207.327416] Key type ._llcrypt unregistered [ 6213.153772] Key type ._llcrypt registered [ 6213.155616] Key type .llcrypt registered [ 6213.190372] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 6219.532628] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6219.991226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6220.002856] alg: No test for adler32 (adler32-zlib) [ 6220.880459] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6220.972901] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6222.551216] Key type lgssc registered [ 6222.925458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6234.104673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6234.110086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.188347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6235.199817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6235.230536] Lustre: lustre-MDT0000: new disk, initializing [ 6235.248951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.254965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6236.496212] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6240.565116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.605663] Lustre: 402162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6240.630737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6240.633139] Lustre: Skipped 1 previous similar message [ 6240.664657] Lustre: lustre-MDT0001: new disk, initializing [ 6240.692540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6240.704702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6240.708837] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6242.138705] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6245.769871] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6247.487993] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6249.567402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6249.671626] Lustre: lustre-OST0000: new disk, initializing [ 6249.673389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6249.675965] Lustre: 404129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6249.699335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6251.616312] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6255.094790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6255.098199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6255.123305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6255.410471] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6256.834957] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6256.910726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6258.351193] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6258.423642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6260.193720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6260.193985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.196528] LustreError: Skipped 1 previous similar message [ 6260.201598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6260.205101] Lustre: Skipped 1 previous similar message [ 6261.136938] Lustre: server umount lustre-OST0000 complete [ 6265.312911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.313625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.317088] Lustre: Skipped 2 previous similar messages [ 6265.321362] Lustre: Skipped 1 previous similar message [ 6268.566278] Lustre: server umount lustre-MDT0000 complete [ 6269.695328] LustreError: 402954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470353 with bad export cookie 4015136931320102166 [ 6269.697240] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6269.700742] LustreError: 402954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6269.811150] Lustre: server umount lustre-MDT0001 complete [ 6272.360679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6272.843807] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:45:56 (1786470356) [ 6274.929831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.073468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.213192] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6278.822479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.211371] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6282.486223] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6284.234094] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6286.467783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6286.573101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.575187] Lustre: Skipped 1 previous similar message [ 6288.491576] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6290.859695] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6291.517554] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6302.664445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6304.223877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.227020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.233145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6305.250021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.293202] Lustre: server umount lustre-OST0000 complete [ 6314.975725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6314.978260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6314.981881] Lustre: Skipped 1 previous similar message [ 6314.983751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6314.985693] Lustre: Skipped 2 previous similar messages [ 6317.780784] Lustre: server umount lustre-MDT0000 complete [ 6319.091280] LustreError: 406565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470402 with bad export cookie 4015136931320103034 [ 6319.093196] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6319.095572] LustreError: 406565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6319.208311] Lustre: server umount lustre-MDT0001 complete [ 6322.815775] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6323.925953] Key type lgssc unregistered [ 6324.053315] LNet: 410328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6324.056329] LNetError: 410328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6324.065473] LNet: Removed LNI 192.168.203.106@tcp [ 6324.395276] Key type .llcrypt unregistered [ 6324.396493] Key type ._llcrypt unregistered [ 6332.163180] Key type ._llcrypt registered [ 6332.164189] Key type .llcrypt registered [ 6332.197660] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6332.304473] Key type .llcrypt unregistered [ 6332.306307] Key type ._llcrypt unregistered [ 6338.955621] Key type ._llcrypt registered [ 6338.956584] Key type .llcrypt registered [ 6338.993342] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6339.310209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6339.316351] alg: No test for adler32 (adler32-zlib) [ 6340.189810] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6340.283567] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6341.863211] Key type lgssc registered [ 6342.222842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6344.905817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6344.915133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.063129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.383541] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6350.093796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6350.206666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.196833] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6354.753061] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6355.425686] LustreError: 412518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.433185] LustreError: 412518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6356.449285] LustreError: 412542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6356.454734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6356.456019] LustreError: 412542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6357.206916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.326753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.720458] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6366.688161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6366.691971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.698993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6367.714068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6367.716805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6367.716930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.871445] Lustre: server umount lustre-OST0000 complete [ 6376.929321] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6376.931638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.935989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6376.938432] Lustre: Skipped 1 previous similar message [ 6379.352617] Lustre: server umount lustre-MDT0000 complete [ 6380.630221] LustreError: 412498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470464 with bad export cookie 10241516355470061742 [ 6380.632969] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6380.634886] LustreError: 412498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6380.760755] Lustre: server umount lustre-MDT0001 complete [ 6384.218324] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6385.273808] Key type lgssc unregistered [ 6385.390379] LNet: 415667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6385.393315] LNetError: 415667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6385.406884] LNet: Removed LNI 192.168.203.106@tcp [ 6385.723509] Key type .llcrypt unregistered [ 6385.724716] Key type ._llcrypt unregistered [ 6394.664043] Key type ._llcrypt registered [ 6394.665203] Key type .llcrypt registered [ 6394.705462] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6395.008526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6395.018991] alg: No test for adler32 (adler32-zlib) [ 6395.878782] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6395.966785] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6397.551145] Key type lgssc registered [ 6397.897255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6400.484815] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6400.492157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.903316] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6421.023317] LustreError: 417034:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 6447.647170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.567856] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6451.983212] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6454.266512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.431179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.454055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6455.810072] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6458.741919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.871834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6460.155682] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6462.456165] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6464.252497] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6468.066163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.072706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6468.075031] Lustre: Skipped 1 previous similar message [ 6472.656125] Lustre: server umount lustre-OST0000 complete [ 6474.207814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6474.210114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.214086] Lustre: Skipped 1 previous similar message [ 6474.216688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.304049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.304499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.081526] Lustre: server umount lustre-MDT0000 complete [ 6481.387686] LustreError: 417041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470565 with bad export cookie 13970950588072470808 [ 6481.392407] LustreError: 417041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6481.393301] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.506895] Lustre: server umount lustre-MDT0001 complete [ 6485.180021] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6486.197451] Key type lgssc unregistered [ 6486.326474] LNet: 420606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6486.328787] LNetError: 420606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6486.337342] LNet: Removed LNI 192.168.203.106@tcp [ 6486.639205] Key type .llcrypt unregistered [ 6486.640442] Key type ._llcrypt unregistered [ 6495.672857] Key type ._llcrypt registered [ 6495.674201] Key type .llcrypt registered [ 6495.710220] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6496.064213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6496.094897] alg: No test for adler32 (adler32-zlib) [ 6496.939849] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6497.022371] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6498.599158] Key type lgssc registered [ 6498.961623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6501.525075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6501.531318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6502.672263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6504.027769] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6506.929515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.044433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6508.406615] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6510.703067] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6512.497130] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6514.740534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.877604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6516.766229] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6519.016722] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6528.233976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6532.577061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6532.579911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.585092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6533.600572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6533.602279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.469470] Lustre: server umount lustre-MDT0000 complete [ 6537.203464] LustreError: 421981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1786470620 with bad export cookie 5807681804324875360 [ 6537.208386] LustreError: 421981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6537.832344] LustreError: 422025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470621 with bad export cookie 5807681804324874737 [ 6537.835691] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.720301] LustreError: 421999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6538.721295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.725306] LustreError: 421999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6538.731959] Lustre: Skipped 1 previous similar message [ 6538.738048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.740213] Lustre: Skipped 3 previous similar messages [ 6542.752179] LustreError: 422018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.752483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6542.758469] LustreError: 422018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6543.953576] Lustre: server umount lustre-OST0000 complete [ 6546.144365] Lustre: server umount lustre-MDT0001 complete [ 6549.723724] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6550.747765] Key type lgssc unregistered [ 6550.877326] LNet: 425707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6550.879964] LNetError: 425707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6550.887283] LNet: Removed LNI 192.168.203.106@tcp [ 6551.223907] Key type .llcrypt unregistered [ 6551.225096] Key type ._llcrypt unregistered [ 6560.258496] Key type ._llcrypt registered [ 6560.259724] Key type .llcrypt registered [ 6560.297871] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6560.583775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6560.592848] alg: No test for adler32 (adler32-zlib) [ 6561.442550] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6561.522405] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6563.095162] Key type lgssc registered [ 6563.475301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6566.100066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6566.105716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.239029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6568.489198] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6571.197576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.320597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6572.648685] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6574.967518] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6576.738952] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6579.000729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.128930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6581.025574] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6583.358322] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6591.397365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6596.580325] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.584740] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6596.587731] Lustre: Skipped 1 previous similar message [ 6599.037515] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 6601.699713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6601.701927] Lustre: Skipped 1 previous similar message [ 6604.150076] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 6609.269130] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.6@tcp (stopping) [ 6609.273056] Lustre: Skipped 2 previous similar messages [ 6609.375142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6609.430087] Lustre: server umount lustre-MDT0001 complete [ 6611.936083] LustreError: 427098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.936593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6611.936636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.936640] Lustre: Skipped 1 previous similar message [ 6611.940793] LustreError: 427098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6617.056552] LustreError: 427099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.236624] Lustre: server umount lustre-OST0000 complete [ 6618.591829] Lustre: server umount lustre-MDT0000 complete [ 6623.081429] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6624.088944] Key type lgssc unregistered [ 6624.211384] LNet: 430804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6624.214241] LNetError: 430804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6624.225326] LNet: Removed LNI 192.168.203.106@tcp [ 6624.507084] Key type .llcrypt unregistered [ 6624.509654] Key type ._llcrypt unregistered [ 6632.623481] Key type ._llcrypt registered [ 6632.624568] Key type .llcrypt registered [ 6632.661142] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 6638.765109] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6639.055018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6639.090105] alg: No test for adler32 (adler32-zlib) [ 6639.944861] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6640.031413] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6641.615154] Key type lgssc registered [ 6642.000071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6654.617491] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6659.596118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6659.603424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6660.679991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6660.688886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6660.720322] Lustre: lustre-MDT0000: new disk, initializing [ 6660.739344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6660.745659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6661.993509] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6667.363180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.452574] Lustre: lustre-OST0000: new disk, initializing [ 6667.454251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6667.455787] Lustre: Skipped 1 previous similar message [ 6667.457787] Lustre: 435807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6667.476859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.197962] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6670.072515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6670.075106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6670.083210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6672.693315] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6677.883348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.928872] Lustre: 434789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6677.958288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6677.998141] Lustre: lustre-MDT0001: new disk, initializing [ 6678.038473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6678.052376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6678.056368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6679.390297] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6682.853810] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6682.921748] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6684.217975] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6686.314452] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6691.297473] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6691.297880] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.300580] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6691.303518] Lustre: Skipped 1 previous similar message [ 6691.307309] Lustre: Skipped 1 previous similar message [ 6696.417733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6696.419720] Lustre: Skipped 2 previous similar messages [ 6701.536921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6701.538579] Lustre: Skipped 2 previous similar messages [ 6702.047126] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6702.107820] Lustre: server umount lustre-MDT0001 complete [ 6711.776867] LustreError: 434796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.781010] LustreError: 434796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6716.383957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6716.386249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.390803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6716.392873] Lustre: Skipped 1 previous similar message [ 6716.896231] LustreError: 434800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6720.466644] Lustre: server umount lustre-OST0000 complete [ 6727.998685] Lustre: server umount lustre-MDT0000 complete [ 6732.313074] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6733.372121] Key type lgssc unregistered [ 6733.502344] LNet: 439567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6733.504911] LNetError: 439567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6733.513330] LNet: Removed LNI 192.168.203.106@tcp [ 6733.836931] Key type .llcrypt unregistered [ 6733.838426] Key type ._llcrypt unregistered [ 6755.157974] Key type ._llcrypt registered [ 6755.159305] Key type .llcrypt registered [ 6755.225858] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6755.419319] Key type .llcrypt unregistered [ 6755.420751] Key type ._llcrypt unregistered [ 6765.403104] Key type ._llcrypt registered [ 6765.404187] Key type .llcrypt registered [ 6765.449846] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6765.873781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6765.932193] alg: No test for adler32 (adler32-zlib) [ 6766.834358] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 6766.931448] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6768.535155] Key type lgssc registered [ 6768.998695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6780.190367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6780.199567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6781.305309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6781.317875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6781.356495] Lustre: lustre-MDT0000: new disk, initializing [ 6781.382956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.392265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6782.932811] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6787.694795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.752663] Lustre: 442921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6787.799519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6787.802749] Lustre: Skipped 1 previous similar message [ 6787.843408] Lustre: lustre-MDT0001: new disk, initializing [ 6787.889331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6787.906272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6787.909819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6789.643101] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6793.787401] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6795.947291] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6798.467784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6798.568452] Lustre: lustre-OST0000: new disk, initializing [ 6798.570603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6798.573153] Lustre: 444888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6798.601207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6800.660408] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6801.790066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6801.794455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6801.832284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6804.747076] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6808.544215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6808.547325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.552701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.129643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6812.129706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6812.131706] Lustre: Skipped 1 previous similar message [ 6812.135290] Lustre: Skipped 1 previous similar message [ 6813.403465] Lustre: server umount lustre-MDT0000 complete [ 6814.763302] LustreError: 442915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470898 with bad export cookie 6316513423150079588 [ 6814.765670] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6814.768734] LustreError: 442915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6814.893337] Lustre: server umount lustre-MDT0001 complete [ 6826.332420] Lustre: server umount lustre-OST0000 complete [ 6829.838111] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:55:12 (1786470912) [ 6832.489337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.643437] LustreError: 446917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6832.651959] LustreError: 446917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6832.669586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.242461] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6837.500179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.626988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6839.191973] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6842.095263] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6844.164954] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6859.490468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6859.576783] Lustre: Found index 0 for lustre-OST0000, updating log [ 6859.639103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.750696] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6864.418813] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6879.077925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6883.295932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6883.299405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6883.303651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6883.305474] Lustre: Skipped 1 previous similar message [ 6887.498890] Lustre: server umount lustre-OST0000 complete [ 6900.013198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.013768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6900.019975] Lustre: Skipped 2 previous similar messages [ 6900.025924] Lustre: Skipped 4 previous similar messages [ 6905.312139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.314840] Lustre: Skipped 1 previous similar message [ 6906.131927] Lustre: server umount lustre-MDT0000 complete [ 6907.447657] LustreError: 446897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786470991 with bad export cookie 6316513423150080974 [ 6907.452392] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6907.452659] LustreError: 446897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6907.569303] Lustre: server umount lustre-MDT0001 complete [ 6910.195730] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:56:33 (1786470993) [ 6927.711301] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [ 7492.840595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.026209] LustreError: 453353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.054060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.770937] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7498.107305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.207841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7498.260631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7500.028796] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7503.205725] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7505.753673] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7522.084726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7522.198224] Lustre: Found index 0 for lustre-OST0000, updating log [ 7522.357331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.033785] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7528.014641] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7530.465614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7530.466261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7530.470562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.864659] Lustre: server umount lustre-OST0000 complete [ 7537.313753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.321595] Lustre: Skipped 1 previous similar message [ 7542.434050] LustreError: 453372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.442699] LustreError: 453372:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7542.455674] Lustre: server umount lustre-MDT0000 complete [ 7543.457278] LustreError: 453353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7543.464055] LustreError: 453353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7544.247736] LustreError: 453335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786471627 with bad export cookie 6316513423150082052 [ 7544.251452] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7544.252782] LustreError: 453335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7544.416677] Lustre: server umount lustre-MDT0001 complete [ 7555.487273] LNet: 1 local NIs in recovery (showing 1): 192.168.203.106@tcp [ 7563.083715] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 7566.621475] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7580.069335] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:07:43 (1786471663) [ 7587.054919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7588.310563] Key type lgssc unregistered [ 7588.440506] LNet: 459969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7588.445909] LNetError: 459969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7652.966975] LNet: Removed LNI 192.168.203.106@tcp [ 7653.441260] Key type .llcrypt unregistered [ 7653.443250] Key type ._llcrypt unregistered [ 7661.865438] Key type ._llcrypt registered [ 7661.866723] Key type .llcrypt registered [ 7661.914128] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7662.447314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7662.467991] alg: No test for adler32 (adler32-zlib) [ 7663.347985] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 7663.460824] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7665.063373] Key type lgssc registered [ 7665.528984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7667.674834] LNet: Added LNI 192.168.203.106@tcp100 [8/256/0/180] [ 7668.125969] LNet: Added LNI 192.168.203.106@tcp101 [8/256/0/180] [ 7668.580935] LNet: Added LNI 192.168.203.106@tcp102 [8/256/0/180] [ 7668.967944] LNet: Added LNI 192.168.203.106@tcp103 [8/256/0/180] [ 7669.418968] LNet: Added LNI 192.168.203.106@tcp104 [8/256/0/180] [ 7669.822252] LNet: Added LNI 192.168.203.106@tcp105 [8/256/0/180] [ 7670.209335] LNet: Added LNI 192.168.203.106@tcp106 [8/256/0/180] [ 7670.632891] LNet: Added LNI 192.168.203.106@tcp107 [8/256/0/180] [ 7671.052848] LNet: Added LNI 192.168.203.106@tcp108 [8/256/0/180] [ 7671.494408] LNet: Added LNI 192.168.203.106@tcp109 [8/256/0/180] [ 7671.938047] LNet: Added LNI 192.168.203.106@tcp110 [8/256/0/180] [ 7672.336943] LNet: Added LNI 192.168.203.106@tcp111 [8/256/0/180] [ 7672.713553] LNet: Added LNI 192.168.203.106@tcp112 [8/256/0/180] [ 7673.144814] LNet: Added LNI 192.168.203.106@tcp113 [8/256/0/180] [ 7673.539757] LNet: Added LNI 192.168.203.106@tcp114 [8/256/0/180] [ 7673.956968] LNet: Added LNI 192.168.203.106@tcp115 [8/256/0/180] [ 7674.356198] LNet: Added LNI 192.168.203.106@tcp116 [8/256/0/180] [ 7674.787961] LNet: Added LNI 192.168.203.106@tcp117 [8/256/0/180] [ 7675.235177] LNet: Added LNI 192.168.203.106@tcp118 [8/256/0/180] [ 7675.674453] LNet: Added LNI 192.168.203.106@tcp119 [8/256/0/180] [ 7676.089698] LNet: Added LNI 192.168.203.106@tcp120 [8/256/0/180] [ 7676.486765] LNet: Added LNI 192.168.203.106@tcp121 [8/256/0/180] [ 7676.918759] LNet: Added LNI 192.168.203.106@tcp122 [8/256/0/180] [ 7677.335126] LNet: Added LNI 192.168.203.106@tcp123 [8/256/0/180] [ 7677.781563] LNet: Added LNI 192.168.203.106@tcp124 [8/256/0/180] [ 7678.216169] LNet: Added LNI 192.168.203.106@tcp125 [8/256/0/180] [ 7678.644883] LNet: Added LNI 192.168.203.106@tcp126 [8/256/0/180] [ 7679.078942] LNet: Added LNI 192.168.203.106@tcp127 [8/256/0/180] [ 7679.516465] LNet: Added LNI 192.168.203.106@tcp128 [8/256/0/180] [ 7679.954143] LNet: Added LNI 192.168.203.106@tcp129 [8/256/0/180] [ 7680.386938] LNet: Added LNI 192.168.203.106@tcp130 [8/256/0/180] [ 7680.786297] LNet: Added LNI 192.168.203.106@tcp131 [8/256/0/180] [ 7681.209143] LNet: Added LNI 192.168.203.106@tcp132 [8/256/0/180] [ 7681.633674] LNet: Added LNI 192.168.203.106@tcp133 [8/256/0/180] [ 7682.061678] LNet: Added LNI 192.168.203.106@tcp134 [8/256/0/180] [ 7682.488496] LNet: Added LNI 192.168.203.106@tcp135 [8/256/0/180] [ 7685.759284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7685.770658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.844245] Lustre: 462965:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [ 7686.892760] LNetError: 460428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7686.908159] LNetError: Protocol error connecting to 192.168.203.106@tcp119 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7686.913070] LNetError: Skipped 3 previous similar messages [ 7686.973990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7686.992588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7687.007309] LNet: 6 local NIs in recovery (showing 6): 192.168.203.106@tcp125, 192.168.203.106@tcp127, 192.168.203.106@tcp126, 192.168.203.106@tcp129, 192.168.203.106@tcp119, 192.168.203.106@tcp128 [ 7687.054767] Lustre: lustre-MDT0000: new disk, initializing [ 7687.104430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7687.122320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7689.058965] LNetError: 462985:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7689.068307] LNetError: 462985:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 19 previous similar messages [ 7689.073918] LNetError: Protocol error connecting to 192.168.203.106@tcp125 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7689.079806] LNetError: Skipped 15 previous similar messages [ 7689.250140] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7693.152644] LNetError: 462987:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7693.160079] LNetError: 462987:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7693.166862] LNetError: Protocol error connecting to 192.168.203.106@tcp125 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7694.651504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.714246] Lustre: 462982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7694.757079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7694.760368] Lustre: Skipped 1 previous similar message [ 7694.800450] Lustre: lustre-MDT0001: new disk, initializing [ 7694.819908] Lustre: 463781:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [ 7694.825506] Lustre: 463781:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7694.892928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.914224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7694.920742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7696.381996] Lustre: 463982:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [ 7696.387196] Lustre: 463982:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7696.764067] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7701.345419] LNetError: 460428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7701.353311] LNetError: Protocol error connecting to 192.168.203.106@tcp125 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7701.548483] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7704.037226] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7706.763756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7706.903768] Lustre: lustre-OST0000: new disk, initializing [ 7706.906265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7706.910445] Lustre: 464953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7706.955190] Lustre: 464958:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [ 7706.959119] Lustre: 464958:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7706.981085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.309053] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7713.572388] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7715.830386] Lustre: 465824:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.106@tcp: can't add NID 192.168.203.106@tcp130: rc = -75 [ 7715.834403] Lustre: 465824:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7715.855046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7715.859356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7715.909161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7717.729895] LNetError: 462992:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7717.734991] LNetError: Protocol error connecting to 192.168.203.106@tcp122 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7720.929338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.931373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.933078] LustreError: Skipped 1 previous similar message [ 7720.933726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7720.937790] Lustre: Skipped 1 previous similar message [ 7722.455493] Lustre: server umount lustre-OST0000 complete [ 7725.536525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7725.542081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7725.552741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7725.556340] Lustre: Skipped 1 previous similar message [ 7725.921440] LNetError: 462991:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7725.927076] LNetError: 462991:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7725.931036] LNetError: Protocol error connecting to 192.168.203.106@tcp121 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7725.936094] LNetError: Skipped 3 previous similar messages [ 7731.169069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.172133] Lustre: Skipped 3 previous similar messages [ 7736.288645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.292162] Lustre: Skipped 1 previous similar message [ 7738.335247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7738.435112] Lustre: server umount lustre-MDT0000 complete [ 7740.009237] LustreError: 462974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786471823 with bad export cookie 12823362640946033450 [ 7740.012359] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.015512] LustreError: 462974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7740.163752] Lustre: server umount lustre-MDT0001 complete [ 7743.328426] LNetError: 462993:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.106 [ 7743.332771] LNetError: 462993:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7743.336594] LNetError: Protocol error connecting to 192.168.203.106@tcp122 at host 192.168.203.106:988: is it running a compatible version of Lustre? [ 7743.343474] LNetError: Skipped 7 previous similar messages [ 7744.469041] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7745.739780] Key type lgssc unregistered [ 7745.912074] LNet: 466870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7745.915096] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7745.918232] LNet: Removed LNI 192.168.203.106@tcp [ 7746.975184] LNet: Removed LNI 192.168.203.106@tcp100 [ 7746.977127] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7746.980568] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7746.984297] LNet: Removed LNI 192.168.203.106@tcp101 [ 7747.999182] LNet: Removed LNI 192.168.203.106@tcp102 [ 7748.002074] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.007020] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7748.010448] LNet: Removed LNI 192.168.203.106@tcp103 [ 7749.023241] LNet: Removed LNI 192.168.203.106@tcp104 [ 7750.047131] LNet: Removed LNI 192.168.203.106@tcp105 [ 7750.049178] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7750.052032] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7750.055354] LNet: Removed LNI 192.168.203.106@tcp106 [ 7751.071687] LNet: Removed LNI 192.168.203.106@tcp107 [ 7752.095130] LNet: Removed LNI 192.168.203.106@tcp108 [ 7753.119256] LNet: Removed LNI 192.168.203.106@tcp109 [ 7754.143140] LNet: Removed LNI 192.168.203.106@tcp110 [ 7754.145559] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7754.148996] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7754.152762] LNet: Removed LNI 192.168.203.106@tcp111 [ 7755.167271] LNet: Removed LNI 192.168.203.106@tcp112 [ 7756.191627] LNet: Removed LNI 192.168.203.106@tcp113 [ 7757.215158] LNet: Removed LNI 192.168.203.106@tcp114 [ 7758.239284] LNet: Removed LNI 192.168.203.106@tcp115 [ 7759.263154] LNet: Removed LNI 192.168.203.106@tcp116 [ 7760.287109] LNet: Removed LNI 192.168.203.106@tcp117 [ 7761.311130] LNet: Removed LNI 192.168.203.106@tcp118 [ 7762.335149] LNet: Removed LNI 192.168.203.106@tcp119 [ 7762.337485] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7762.341889] LNetError: 466870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7762.347310] LNet: Removed LNI 192.168.203.106@tcp120 [ 7763.359154] LNet: Removed LNI 192.168.203.106@tcp121 [ 7764.383129] LNet: Removed LNI 192.168.203.106@tcp122 [ 7765.407126] LNet: Removed LNI 192.168.203.106@tcp123 [ 7766.431120] LNet: Removed LNI 192.168.203.106@tcp124 [ 7767.455154] LNet: Removed LNI 192.168.203.106@tcp125 [ 7768.479151] LNet: Removed LNI 192.168.203.106@tcp126 [ 7769.503280] LNet: Removed LNI 192.168.203.106@tcp127 [ 7770.527163] LNet: Removed LNI 192.168.203.106@tcp128 [ 7771.551240] LNet: Removed LNI 192.168.203.106@tcp129 [ 7771.553363] LNet: Removed LNI 192.168.203.106@tcp130 [ 7771.555464] LNet: Removed LNI 192.168.203.106@tcp131 [ 7771.558178] LNet: Removed LNI 192.168.203.106@tcp132 [ 7771.559891] LNet: Removed LNI 192.168.203.106@tcp133 [ 7771.562206] LNet: Removed LNI 192.168.203.106@tcp134 [ 7771.569776] LNet: Removed LNI 192.168.203.106@tcp135 [ 7771.899989] Key type .llcrypt unregistered [ 7771.901590] Key type ._llcrypt unregistered [ 7781.391725] Key type ._llcrypt registered [ 7781.393251] Key type .llcrypt registered [ 7781.436971] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 7788.164438] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7788.619467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7788.640061] alg: No test for adler32 (adler32-zlib) [ 7789.505729] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 7789.609315] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7791.207137] Key type lgssc registered [ 7791.631799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7805.443491] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:11:28 (1786471888) [ 7812.378725] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7813.507351] Key type lgssc unregistered [ 7813.642412] LNet: 470934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7813.646100] LNetError: 470934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7813.653365] LNet: Removed LNI 192.168.203.106@tcp [ 7813.979411] Key type .llcrypt unregistered [ 7813.981218] Key type ._llcrypt unregistered [ 7821.135125] Key type ._llcrypt registered [ 7821.137107] Key type .llcrypt registered [ 7821.175536] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7821.625987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7821.636400] alg: No test for adler32 (adler32-zlib) [ 7822.493892] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 7822.589242] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7824.175181] Key type lgssc registered [ 7824.583882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7828.142682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7828.149775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.258687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7829.270068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7829.306392] Lustre: lustre-MDT0000: new disk, initializing [ 7829.327358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7829.334262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7830.768407] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7835.350661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.396653] Lustre: 472122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7835.423309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7835.425495] Lustre: Skipped 1 previous similar message [ 7835.461829] Lustre: lustre-MDT0001: new disk, initializing [ 7835.499312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7835.512314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7835.516154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7837.016889] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7841.260976] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7843.591298] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7846.341115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7846.452769] Lustre: lustre-OST0000: new disk, initializing [ 7846.455336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7846.459265] Lustre: 474091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7846.494031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.576296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7846.581199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7846.613543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7848.776716] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7853.167281] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7854.644550] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing lnet_if_list [ 7856.643076] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7856.658464] LNet: Added LNI 192.168.203.106@tcp10 [8/256/0/180] [ 7858.849467] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7858.864676] LNet: Added LNI 192.168.203.106@tcp11 [8/256/0/180] [ 7860.611327] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7860.626949] LNet: Added LNI 192.168.203.106@tcp12 [8/256/0/180] [ 7862.416323] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7862.433675] LNet: Added LNI 192.168.203.106@tcp13 [8/256/0/180] [ 7864.130783] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7864.148542] LNet: Added LNI 192.168.203.106@tcp14 [8/256/0/180] [ 7865.919736] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7865.935502] LNet: Added LNI 192.168.203.106@tcp15 [8/256/0/180] [ 7867.683789] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7867.697225] LNet: Added LNI 192.168.203.106@tcp16 [8/256/0/180] [ 7869.461628] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7869.478628] LNet: Added LNI 192.168.203.106@tcp17 [8/256/0/180] [ 7871.327932] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7871.343097] LNet: Added LNI 192.168.203.106@tcp18 [8/256/0/180] [ 7873.291623] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7873.304035] LNet: Added LNI 192.168.203.106@tcp19 [8/256/0/180] [ 7875.208685] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7875.221585] LNet: Added LNI 192.168.203.106@tcp20 [8/256/0/180] [ 7877.150558] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7877.163419] LNet: Added LNI 192.168.203.106@tcp21 [8/256/0/180] [ 7879.311345] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7879.326172] LNet: Added LNI 192.168.203.106@tcp22 [8/256/0/180] [ 7881.512087] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7881.525741] LNet: Added LNI 192.168.203.106@tcp23 [8/256/0/180] [ 7883.759340] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7883.779246] LNet: Added LNI 192.168.203.106@tcp24 [8/256/0/180] [ 7885.970733] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7885.983961] LNet: Added LNI 192.168.203.106@tcp25 [8/256/0/180] [ 7888.382860] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7888.395991] LNet: Added LNI 192.168.203.106@tcp26 [8/256/0/180] [ 7890.695118] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7890.708898] LNet: Added LNI 192.168.203.106@tcp27 [8/256/0/180] [ 7892.999197] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7893.015452] LNet: Added LNI 192.168.203.106@tcp28 [8/256/0/180] [ 7895.441138] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7895.457456] LNet: Added LNI 192.168.203.106@tcp29 [8/256/0/180] [ 7898.029344] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7898.044180] LNet: Added LNI 192.168.203.106@tcp30 [8/256/0/180] [ 7900.422955] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7900.435047] LNet: Added LNI 192.168.203.106@tcp31 [8/256/0/180] [ 7903.007686] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7903.022077] LNet: Added LNI 192.168.203.106@tcp32 [8/256/0/180] [ 7905.649604] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7905.665750] LNet: Added LNI 192.168.203.106@tcp33 [8/256/0/180] [ 7908.227844] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7908.241982] LNet: Added LNI 192.168.203.106@tcp34 [8/256/0/180] [ 7910.795226] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7910.809165] LNet: Added LNI 192.168.203.106@tcp35 [8/256/0/180] [ 7913.551927] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7913.570367] LNet: Added LNI 192.168.203.106@tcp36 [8/256/0/180] [ 7916.494655] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7916.510462] LNet: Added LNI 192.168.203.106@tcp37 [8/256/0/180] [ 7919.415646] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7919.431813] LNet: Added LNI 192.168.203.106@tcp38 [8/256/0/180] [ 7922.315904] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7922.334626] LNet: Added LNI 192.168.203.106@tcp39 [8/256/0/180] [ 7925.475437] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7925.490653] LNet: Added LNI 192.168.203.106@tcp40 [8/256/0/180] [ 7928.667371] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7928.685051] LNet: Added LNI 192.168.203.106@tcp41 [8/256/0/180] [ 7931.767904] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7931.785565] LNet: Added LNI 192.168.203.106@tcp42 [8/256/0/180] [ 7934.965449] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7934.981645] LNet: Added LNI 192.168.203.106@tcp43 [8/256/0/180] [ 7938.329235] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7938.345842] LNet: Added LNI 192.168.203.106@tcp44 [8/256/0/180] [ 7941.865328] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7941.880625] LNet: Added LNI 192.168.203.106@tcp45 [8/256/0/180] [ 7945.173842] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7945.190137] LNet: Added LNI 192.168.203.106@tcp46 [8/256/0/180] [ 7948.742203] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7948.757767] LNet: Added LNI 192.168.203.106@tcp47 [8/256/0/180] [ 7952.411672] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7952.427493] LNet: Added LNI 192.168.203.106@tcp48 [8/256/0/180] [ 7956.158478] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7956.172374] LNet: Added LNI 192.168.203.106@tcp49 [8/256/0/180] [ 7959.699726] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7959.715576] LNet: Added LNI 192.168.203.106@tcp50 [8/256/0/180] [ 7968.736297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7968.739798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.745258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.761098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.764328] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7974.303298] Lustre: server umount lustre-MDT0000 complete [ 7975.698537] LustreError: 472115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786472059 with bad export cookie 12339588598036435163 [ 7975.700933] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.703770] LustreError: 472115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7975.833991] Lustre: server umount lustre-MDT0001 complete [ 7983.896620] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7987.157594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.383135] Lustre: 471396:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786472063/real 1786472063] req@000000008184dbce x1873251565398656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786472079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7996.383261] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.393544] Lustre: 471396:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7996.399785] Lustre: Skipped 1 previous similar message [ 7999.455164] Lustre: 471398:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786472067/real 1786472067] req@0000000067d19dc9 x1873251565398912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786472083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8000.479365] LustreError: 471394:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005ef445ef x1873251565399680/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8000.575443] LustreError: 474083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8000.584483] LustreError: 474083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8000.591433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8001.631108] Lustre: 471397:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786472069/real 1786472069] req@000000004fe6d743 x1873251565399168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786472085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8001.635301] LustreError: 474084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.044102] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8004.640881] LustreError: 474083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.240379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8005.383170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.439572] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8006.904293] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8007.455996] Lustre: 471395:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786472074/real 1786472074] req@00000000fd05093c x1873251565399296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786472090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8009.621206] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8011.711241] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8020.961695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8020.962257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.966245] LustreError: Skipped 1 previous similar message [ 8020.966996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8020.972321] Lustre: Skipped 1 previous similar message [ 8020.979263] Lustre: Skipped 3 previous similar messages [ 8025.947568] Lustre: server umount lustre-OST0000 complete [ 8031.201130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8031.205117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.211066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8031.213293] Lustre: Skipped 2 previous similar messages [ 8033.513319] Lustre: server umount lustre-MDT0000 complete [ 8034.995336] LustreError: 474090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786472118 with bad export cookie 12339588598036437312 [ 8034.999069] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8035.001364] LustreError: 474090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8035.128772] Lustre: server umount lustre-MDT0001 complete [ 8039.299107] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8040.434317] Key type lgssc unregistered [ 8040.597255] LNet: 501054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8040.601113] LNetError: 501054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8040.604131] LNet: Removed LNI 192.168.203.106@tcp [ 8040.605813] LNet: Removed LNI 192.168.203.106@tcp10 [ 8040.607504] LNet: Removed LNI 192.168.203.106@tcp11 [ 8040.608932] LNet: Removed LNI 192.168.203.106@tcp12 [ 8040.610410] LNet: Removed LNI 192.168.203.106@tcp13 [ 8040.611894] LNet: Removed LNI 192.168.203.106@tcp14 [ 8040.613461] LNet: Removed LNI 192.168.203.106@tcp15 [ 8040.615438] LNet: Removed LNI 192.168.203.106@tcp16 [ 8040.617450] LNet: Removed LNI 192.168.203.106@tcp17 [ 8040.619174] LNet: Removed LNI 192.168.203.106@tcp18 [ 8040.620665] LNet: Removed LNI 192.168.203.106@tcp19 [ 8040.622405] LNet: Removed LNI 192.168.203.106@tcp20 [ 8040.624104] LNet: Removed LNI 192.168.203.106@tcp21 [ 8040.625627] LNet: Removed LNI 192.168.203.106@tcp22 [ 8040.627459] LNet: Removed LNI 192.168.203.106@tcp23 [ 8040.628889] LNet: Removed LNI 192.168.203.106@tcp24 [ 8040.630210] LNet: Removed LNI 192.168.203.106@tcp25 [ 8040.632022] LNet: Removed LNI 192.168.203.106@tcp26 [ 8040.633600] LNet: Removed LNI 192.168.203.106@tcp27 [ 8040.635215] LNet: Removed LNI 192.168.203.106@tcp28 [ 8040.637174] LNet: Removed LNI 192.168.203.106@tcp29 [ 8040.639186] LNet: Removed LNI 192.168.203.106@tcp30 [ 8040.640835] LNet: Removed LNI 192.168.203.106@tcp31 [ 8040.642370] LNet: Removed LNI 192.168.203.106@tcp32 [ 8040.644108] LNet: Removed LNI 192.168.203.106@tcp33 [ 8040.645720] LNet: Removed LNI 192.168.203.106@tcp34 [ 8040.647166] LNet: Removed LNI 192.168.203.106@tcp35 [ 8040.648477] LNet: Removed LNI 192.168.203.106@tcp36 [ 8040.650098] LNet: Removed LNI 192.168.203.106@tcp37 [ 8040.651934] LNet: Removed LNI 192.168.203.106@tcp38 [ 8040.653810] LNet: Removed LNI 192.168.203.106@tcp39 [ 8040.655793] LNet: Removed LNI 192.168.203.106@tcp40 [ 8040.657407] LNet: Removed LNI 192.168.203.106@tcp41 [ 8040.659384] LNet: Removed LNI 192.168.203.106@tcp42 [ 8040.661307] LNet: Removed LNI 192.168.203.106@tcp43 [ 8040.663423] LNet: Removed LNI 192.168.203.106@tcp44 [ 8040.665208] LNet: Removed LNI 192.168.203.106@tcp45 [ 8040.667036] LNet: Removed LNI 192.168.203.106@tcp46 [ 8040.668798] LNet: Removed LNI 192.168.203.106@tcp47 [ 8040.670772] LNet: Removed LNI 192.168.203.106@tcp48 [ 8040.673174] LNet: Removed LNI 192.168.203.106@tcp49 [ 8040.684304] LNet: Removed LNI 192.168.203.106@tcp50 [ 8041.003091] Key type .llcrypt unregistered [ 8041.004729] Key type ._llcrypt unregistered [ 8048.541964] Key type ._llcrypt registered [ 8048.543466] Key type .llcrypt registered [ 8048.598884] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8055.715788] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8056.179138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8056.191330] alg: No test for adler32 (adler32-zlib) [ 8057.058494] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 8057.160853] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8058.751239] Key type lgssc registered [ 8059.169572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8071.869020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8071.880335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.966408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8072.978928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8073.016902] Lustre: lustre-MDT0000: new disk, initializing [ 8073.043335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8073.055084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8074.500979] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8079.293918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8079.344871] Lustre: 504222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8079.374990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8079.377778] Lustre: Skipped 1 previous similar message [ 8079.417562] Lustre: lustre-MDT0001: new disk, initializing [ 8079.460228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8079.475793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8079.479774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8081.123559] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8085.636151] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8087.996237] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8091.075327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8091.172734] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8091.210105] Lustre: lustre-OST0000: new disk, initializing [ 8091.212721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8091.216839] Lustre: 506239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8091.253834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8093.624606] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8097.895897] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8098.817907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8098.822689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8098.869733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8103.906157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8103.907437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.909341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.918229] Lustre: Skipped 1 previous similar message [ 8106.782953] Lustre: server umount lustre-OST0000 complete [ 8109.024677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8109.025604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8109.030930] Lustre: Skipped 1 previous similar message [ 8109.033879] Lustre: Skipped 1 previous similar message [ 8114.144391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8114.147494] Lustre: Skipped 2 previous similar messages [ 8114.533520] Lustre: server umount lustre-MDT0000 complete [ 8116.069321] LustreError: 504215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786472199 with bad export cookie 13844065823411826434 [ 8116.072360] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8116.074617] LustreError: 504215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8116.215910] Lustre: server umount lustre-MDT0001 complete [ 8120.460485] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8121.639327] Key type lgssc unregistered [ 8121.777535] LNet: 508146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8121.780765] LNetError: 508146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8121.789449] LNet: Removed LNI 192.168.203.106@tcp [ 8122.162734] Key type .llcrypt unregistered [ 8122.164279] Key type ._llcrypt unregistered [ 8131.157737] Key type ._llcrypt registered [ 8131.159333] Key type .llcrypt registered [ 8131.201185] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8131.675357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8131.710408] alg: No test for adler32 (adler32-zlib) [ 8132.585118] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 8132.689732] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8134.279178] Key type lgssc registered [ 8134.739452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8137.684596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8137.693995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.866464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8140.426176] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8143.681412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.832439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.415692] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8148.177124] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8150.249915] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8152.756212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8152.890913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.051426] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8157.700884] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8172.216587] loop0: detected capacity change from 0 to 204800000 [ 8172.283596] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8172.289741] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8172.296624] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8174.655541] loop0: detected capacity change from 0 to 204800000 [ 8174.714357] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8174.719772] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8174.726234] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8176.679873] loop0: detected capacity change from 0 to 204800000 [ 8178.799772] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8178.857499] Lustre: fs15246-OST0000: new disk, initializing [ 8178.860360] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8178.864164] Lustre: 513007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8178.888809] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8181.027783] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8185.344251] loop1: detected capacity change from 0 to 204800000 [ 8187.392132] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8187.432633] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8187.460105] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8187.499178] Lustre: fs15246-MDT0000: new disk, initializing [ 8187.566286] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.579892] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8187.619377] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8188.639781] Lustre: 514129:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004ab6e95c x1873251890396672/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786472288 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8189.125409] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8197.727264] LustreError: 514129:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8197.731316] LustreError: 514129:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8197.735108] LustreError: 514129:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8200.163734] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.169065] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8202.783352] Lustre: server umount fs15246-MDT0000 complete [ 8205.286366] LustreError: 509388:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.293052] LustreError: 509388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8206.377618] Lustre: server umount fs15246-OST0000 complete [ 8210.401090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8210.401970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.404866] LustreError: Skipped 1 previous similar message [ 8210.410758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8210.415569] Lustre: Skipped 1 previous similar message [ 8215.522895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8215.525567] Lustre: Skipped 1 previous similar message [ 8216.346383] Lustre: server umount lustre-OST0000 complete [ 8220.642311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8220.642315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.642328] Lustre: Skipped 1 previous similar message [ 8220.645844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.654255] Lustre: Skipped 1 previous similar message [ 8223.892621] Lustre: server umount lustre-MDT0000 complete [ 8225.169776] LustreError: 509376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786472308 with bad export cookie 15444327009507976318 [ 8225.173151] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8225.175622] LustreError: 509376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8225.292495] Lustre: server umount lustre-MDT0001 complete [ 8229.014552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8230.041795] Key type lgssc unregistered [ 8230.179444] LNet: 517223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8230.182793] LNetError: 517223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8230.194324] LNet: Removed LNI 192.168.203.106@tcp [ 8230.534225] Key type .llcrypt unregistered [ 8230.536067] Key type ._llcrypt unregistered [ 8238.243829] Key type ._llcrypt registered [ 8238.245201] Key type .llcrypt registered [ 8238.278446] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8238.609819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.639291] alg: No test for adler32 (adler32-zlib) [ 8239.489612] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 8239.575405] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 8241.151162] Key type lgssc registered [ 8241.513544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8254.200332] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8257.315102] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8269.086977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8269.091802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8270.173855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8270.186309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8270.223766] Lustre: lustre-MDT0000: new disk, initializing [ 8270.251722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8270.258359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8271.614552] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8275.769456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.809977] Lustre: 521657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8275.835225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8275.837693] Lustre: Skipped 1 previous similar message [ 8275.872428] Lustre: lustre-MDT0001: new disk, initializing [ 8275.907671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8275.922599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8275.926564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8277.269281] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8280.999467] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8282.955270] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8285.219841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8285.322375] Lustre: lustre-OST0000: new disk, initializing [ 8285.324152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8285.327408] Lustre: 523626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8285.359801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.862280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8285.866263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8285.908979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8287.299656] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 8291.162580] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8292.687857] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8292.768943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8294.210480] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8294.286611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8296.418481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8296.419341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.429055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8296.432104] Lustre: Skipped 1 previous similar message [ 8301.537551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8301.540063] Lustre: Skipped 1 previous similar message [ 8306.658792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8306.661108] Lustre: Skipped 1 previous similar message [ 8309.215069] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8309.268418] Lustre: server umount lustre-OST0000 complete [ 8311.776775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.778192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8311.778214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8311.778223] LustreError: Skipped 1 previous similar message [ 8311.781216] Lustre: Skipped 2 previous similar messages [ 8316.756295] Lustre: server umount lustre-MDT0000 complete [ 8317.969122] LustreError: 521650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786472401 with bad export cookie 2357717489691688698 [ 8317.971607] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.972891] LustreError: 521650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8318.094140] Lustre: server umount lustre-MDT0001 complete [ 8326.837382] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 8330.495579] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 8340.944188] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8172 sec ========== 14:20:24 (1786472424) [ 8341.477037] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:20:24 (1786472424) === [ 8342.567153] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:20:25 (1786472425) === [ 8356.319675] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 8357.405628] Key type lgssc unregistered [ 8357.541283] LNet: 530257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8357.544534] LNetError: 530257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8357.551411] LNet: Removed LNI 192.168.203.106@tcp [ 8357.875974] Key type .llcrypt unregistered [ 8357.877631] Key type ._llcrypt unregistered