[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488749560 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003205] x2apic enabled [ 0.005003] Switched APIC routing to physical x2apic. [ 0.006014] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.012035] LSM: Security Framework initializing [ 0.013076] Yama: becoming mindful. [ 0.014057] SELinux: Initializing. [ 0.015101] *** VALIDATE selinux *** [ 0.024826] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028612] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029199] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031157] *** VALIDATE tmpfs *** [ 0.033254] *** VALIDATE proc *** [ 0.034302] *** VALIDATE cgroup *** [ 0.035015] *** VALIDATE cgroup2 *** [ 0.036319] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040016] Speculative Store Bypass: Vulnerable [ 0.043360] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.045213] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046830] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047030] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050018] ... value mask: 0000ffffffffffff [ 0.051020] ... max period: 00007fffffffffff [ 0.052022] ... fixed-purpose events: 3 [ 0.053021] ... event mask: 000000070000000f [ 0.055306] rcu: Hierarchical SRCU implementation. [ 0.057796] smp: Bringing up secondary CPUs ... [ 0.058730] x86: Booting SMP configuration: [ 0.059039] .... node #0, CPUs: #1 #2 #3 [ 0.062593] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142773] node 0 deferred pages initialised in 76ms [ 0.147010] devtmpfs: initialized [ 0.148232] x86/mm: Memory block size: 128MB [ 0.150654] gcov: version magic: 0x41383552 [ 0.152317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154320] pinctrl core: initialized pinctrl subsystem [ 0.155275] [ 0.155981] ************************************************************* [ 0.156024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157020] ** ** [ 0.158016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159016] ** ** [ 0.160020] ** This means that this kernel is built to expose internal ** [ 0.161017] ** IOMMU data structures, which may compromise security on ** [ 0.162015] ** your system. ** [ 0.163019] ** ** [ 0.164016] ** If you see this message and you are not debugging the ** [ 0.165019] ** kernel, report this immediately to your vendor! ** [ 0.166017] ** ** [ 0.167017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168018] ************************************************************* [ 0.169754] NET: Registered protocol family 16 [ 0.170512] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174021] cpuidle: using governor menu [ 0.178397] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182862] PCI: Using configuration type 1 for base access [ 0.185148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204034] cryptd: max_cpu_qlen set to 1000 [ 0.207260] ACPI: Added _OSI(Module Device) [ 0.209027] ACPI: Added _OSI(Processor Device) [ 0.211020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218275] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227629] ACPI: Interpreter enabled [ 0.228066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229023] ACPI: Using IOAPIC for interrupt routing [ 0.232106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238457] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249454] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.252060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.255027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266191] acpiphp: Slot [2] registered [ 0.268215] acpiphp: Slot [5] registered [ 0.269392] acpiphp: Slot [6] registered [ 0.271303] acpiphp: Slot [7] registered [ 0.273165] acpiphp: Slot [8] registered [ 0.275207] acpiphp: Slot [9] registered [ 0.276163] acpiphp: Slot [10] registered [ 0.278152] acpiphp: Slot [3] registered [ 0.280191] acpiphp: Slot [4] registered [ 0.281136] acpiphp: Slot [11] registered [ 0.283146] acpiphp: Slot [12] registered [ 0.285118] acpiphp: Slot [13] registered [ 0.286219] acpiphp: Slot [14] registered [ 0.288139] acpiphp: Slot [15] registered [ 0.290136] acpiphp: Slot [16] registered [ 0.291186] acpiphp: Slot [17] registered [ 0.293128] acpiphp: Slot [18] registered [ 0.295141] acpiphp: Slot [19] registered [ 0.296179] acpiphp: Slot [20] registered [ 0.298168] acpiphp: Slot [21] registered [ 0.299164] acpiphp: Slot [22] registered [ 0.301121] acpiphp: Slot [23] registered [ 0.304172] acpiphp: Slot [24] registered [ 0.306203] acpiphp: Slot [25] registered [ 0.307134] acpiphp: Slot [26] registered [ 0.309149] acpiphp: Slot [27] registered [ 0.310224] acpiphp: Slot [28] registered [ 0.312135] acpiphp: Slot [29] registered [ 0.313120] acpiphp: Slot [30] registered [ 0.315140] acpiphp: Slot [31] registered [ 0.316111] PCI host bridge to bus 0000:00 [ 0.318028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.327029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331237] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334471] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338406] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348732] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.352480] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364278] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367896] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370220] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375739] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397881] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.401917] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.408466] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.423021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.467022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.486021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.501112] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.510019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.537022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546737] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.579016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.592000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.607020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.627031] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.637966] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.647022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.685028] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.687428] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.689351] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.691578] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.694369] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.700109] iommu: Default domain type: Passthrough [ 0.701818] SCSI subsystem initialized [ 0.702201] ACPI: bus type USB registered [ 0.703139] usbcore: registered new interface driver usbfs [ 0.706145] usbcore: registered new interface driver hub [ 0.709155] usbcore: registered new device driver usb [ 0.711243] pps_core: LinuxPPS API ver. 1 registered [ 0.713018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.716091] PTP clock support registered [ 0.718175] EDAC MC: Ver: 3.0.0 [ 0.720130] PCI: Using ACPI for IRQ routing [ 0.721912] NetLabel: Initializing [ 0.724015] NetLabel: domain hash size = 128 [ 0.726015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729111] NetLabel: unlabeled traffic allowed by default [ 0.733174] vgaarb: loaded [ 0.735307] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744182] clocksource: Switched to clocksource kvm-clock [ 0.858991] VFS: Disk quotas dquot_6.6.0 [ 0.860655] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863304] *** VALIDATE ramfs *** [ 0.864744] *** VALIDATE hugetlbfs *** [ 0.866452] pnp: PnP ACPI init [ 0.869108] pnp: PnP ACPI: found 6 devices [ 0.888837] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.891230] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.893661] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.895993] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.898538] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.900988] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.904726] NET: Registered protocol family 2 [ 0.908185] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.914115] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.918360] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.924122] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.927651] TCP: Hash tables configured (established 65536 bind 65536) [ 0.930749] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.933720] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.936610] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.939461] NET: Registered protocol family 1 [ 0.943901] RPC: Registered named UNIX socket transport module. [ 0.946565] RPC: Registered udp transport module. [ 0.948627] RPC: Registered tcp transport module. [ 0.950724] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.953647] NET: Registered protocol family 44 [ 0.955529] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.957895] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.960378] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962925] PCI: CLS 0 bytes, default 64 [ 0.964881] Unpacking initramfs... [ 2.354216] debug: unmapping init [mem 0xffffa03efcc54000-0xffffa03efffbffff] [ 2.358132] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.360809] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.364572] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.867489] Initialise system trusted keyrings [ 2.869266] Key type blacklist registered [ 2.871284] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.880548] zbud: loaded [ 2.883703] *** VALIDATE nfs *** [ 2.884909] *** VALIDATE nfs4 *** [ 2.886633] pstore: using deflate compression [ 2.890554] Platform Keyring initialized [ 3.002862] NET: Registered protocol family 38 [ 3.004748] Key type asymmetric registered [ 3.006280] Asymmetric key parser 'x509' registered [ 3.007930] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.010717] io scheduler mq-deadline registered [ 3.012392] io scheduler kyber registered [ 3.013975] io scheduler bfq registered [ 3.015884] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.019285] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.022644] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.025989] ACPI: Power Button [PWRF] [ 3.032679] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.043639] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.059232] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.071609] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.089145] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.120000] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.150111] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.155783] Non-volatile memory driver v1.3 [ 3.157920] Linux agpgart interface v0.103 [ 3.193773] virtio_blk virtio1: [vda] 136576 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.198492] vda: detected capacity change from 0 to 69926912 [ 3.221959] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.225217] vdb: detected capacity change from 0 to 1073741824 [ 3.249502] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.252249] vdc: detected capacity change from 0 to 2621440000 [ 3.272446] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.275886] vdd: detected capacity change from 0 to 2621440000 [ 3.294263] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.297256] vde: detected capacity change from 0 to 4294967296 [ 3.314183] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.317833] vdf: detected capacity change from 0 to 4294967296 [ 3.327950] libphy: Fixed MDIO Bus: probed [ 3.337294] usbcore: registered new interface driver usbserial_generic [ 3.340181] usbserial: USB Serial support registered for generic [ 3.343173] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348422] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.349908] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.352395] mousedev: PS/2 mouse device common for all mice [ 3.355302] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.357328] rtc_cmos 00:05: RTC can wake from S4 [ 3.364076] rtc_cmos 00:05: registered as rtc0 [ 3.366814] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.369913] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.371171] intel_pstate: CPU model not supported [ 3.374156] hid: raw HID events driver (C) Jiri Kosina [ 3.379698] usbcore: registered new interface driver usbhid [ 3.382182] usbhid: USB HID core driver [ 3.383615] drop_monitor: Initializing network drop monitor service [ 3.386384] Initializing XFRM netlink socket [ 3.386931] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.388498] NET: Registered protocol family 10 [ 3.395431] Segment Routing with IPv6 [ 3.396728] NET: Registered protocol family 17 [ 3.398520] mpls_gso: MPLS GSO support [ 3.402903] RAS: Correctable Errors collector initialized. [ 3.404639] AVX version of gcm_enc/dec engaged. [ 3.406133] AES CTR mode by8 optimization enabled [ 3.484327] sched_clock: Marking stable (3484303556, 0)->(4444258548, -959954992) [ 3.488092] registered taskstats version 1 [ 3.489771] Loading compiled-in X.509 certificates [ 3.491585] zswap: loaded using pool lzo/zbud [ 3.512521] Key type big_key registered [ 3.523698] Key type encrypted registered [ 3.526183] ima: No TPM chip found, activating TPM-bypass! [ 3.528269] ima: Allocated hash algorithm: sha1 [ 3.529602] ima: No architecture policies found [ 3.530900] evm: Initialising EVM extended attributes: [ 3.532508] evm: security.selinux [ 3.533493] evm: security.ima [ 3.534674] evm: security.capability [ 3.536917] evm: HMAC attrs: 0x1 [ 3.539303] rtc_cmos 00:05: setting system clock to 2026-06-12 04:25:09 UTC (1781238309) [ 3.545626] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.548443] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 3.556152] Write protecting the kernel read-only data: 28672k [ 3.559677] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.562044] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 3.599810] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.606351] systemd[1]: Detected virtualization kvm. [ 3.607816] systemd[1]: Detected architecture x86-64. [ 3.609422] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.636228] systemd[1]: No hostname configured. [ 3.638774] systemd[1]: Set hostname to . [ 3.641193] random: systemd: uninitialized urandom read (16 bytes read) [ 3.644062] systemd[1]: Initializing machine ID from random generator. [ 3.773751] random: systemd: uninitialized urandom read (16 bytes read) [ 3.776985] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.782463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.785653] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.791053] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.432276] device-mapper: uevent: version 1.0.3 [ 4.434107] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.124165] random: fast init done [ 5.131263] virtio_net virtio0 ens2: renamed from eth0 [ 5.205524] scsi host0: ata_piix [ 5.214159] scsi host1: ata_piix [ 5.215948] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.218905] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.416586] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.058511] random: crng init done [ 10.060535] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.394630] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.960622] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.812324] SELinux: Disabled at runtime. [ 12.961032] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.981852] systemd[1]: Detected virtualization kvm. [ 12.986337] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.750770] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.754548] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.769873] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.775313] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.798331] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.823540] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.844688] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slic[ 14.976836] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e User and Session Slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.251892] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.353736] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.386660] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.712435] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.756665] EDAC sbridge: Ver: 1.1.2 [ 21.958661] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 22.597297] NFS: Registering the id_resolver key type [ 22.600470] Key type id_resolver registered [ 22.602615] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg313-server login: [ 84.492972] libcfs: loading out-of-tree module taints kernel. [ 84.538290] Key type ._llcrypt registered [ 84.540497] Key type .llcrypt registered [ 84.653839] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 102.774453] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 104.276918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 104.294130] alg: No test for adler32 (adler32-zlib) [ 105.633956] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 106.492201] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 108.239236] Key type lgssc registered [ 109.855351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.487479] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.532296] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 161.282583] hrtimer: interrupt took 3009616 ns [ 172.922944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.975241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.365322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.404557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.462730] Lustre: lustre-MDT0000: new disk, initializing [ 174.533783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.554926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 179.764660] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.792838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.933506] Lustre: 6508:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.959996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.966753] Lustre: Skipped 1 previous similar message [ 196.104281] Lustre: lustre-MDT0001: new disk, initializing [ 196.187686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.210378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.228539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 200.802742] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.811364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.579408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.795408] Lustre: lustre-OST0000: new disk, initializing [ 214.800092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.807146] Lustre: 8411:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.881370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.356881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.365215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.504882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.084860] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.190937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.356738] Lustre: lustre-OST0001: new disk, initializing [ 235.360580] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 235.373720] Lustre: 9467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.440165] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.419321] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.242132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.250360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.308443] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 251.989176] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 256.904285] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 262.977711] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing check_logdir /tmp/testlogs/ [ 267.016169] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing yml_node [ 274.156081] Lustre: DEBUG MARKER: Client: 2.17.53.78 [ 276.514509] Lustre: DEBUG MARKER: MDS: 2.17.53.78 [ 279.292293] Lustre: DEBUG MARKER: OSS: 2.17.53.78 [ 280.835649] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 00:29:44 EDT 2026 [ 299.251453] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 301.059855] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.896430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.897433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.911144] Lustre: Skipped 1 previous similar message [ 316.928144] Lustre: Skipped 3 previous similar messages [ 322.019671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.024576] Lustre: Skipped 3 previous similar messages [ 322.565663] Lustre: server umount lustre-MDT0000 complete [ 325.634614] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238631 with bad export cookie 7515054209067327797 [ 325.641342] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.642708] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 327.136043] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.147707] Lustre: Skipped 2 previous similar messages [ 327.151686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.150929] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.158250] Lustre: Skipped 2 previous similar messages [ 332.058558] Lustre: server umount lustre-MDT0001 complete [ 336.062714] Lustre: server umount lustre-OST0000 complete [ 339.482903] Lustre: server umount lustre-OST0001 complete [ 345.469878] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 354.514636] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 389.556755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.889706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.938641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.036675] Lustre: lustre-MDT0000: new disk, initializing [ 390.137617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.166588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 394.120708] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 404.103032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.260870] Lustre: 14984:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.328828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.336504] Lustre: Skipped 1 previous similar message [ 404.439742] Lustre: lustre-MDT0001: new disk, initializing [ 404.561725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.627373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.650861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.230360] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 416.416407] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 420.380437] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 426.931379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.211110] Lustre: lustre-OST0000: new disk, initializing [ 427.226818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.237194] Lustre: 16726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.366874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.602589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 428.609651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 428.752332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.474740] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 439.536577] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 443.891223] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.156620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.428974] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.661395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 454.624703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.627624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.629375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.643458] Lustre: Skipped 2 previous similar messages [ 456.914765] Lustre: server umount lustre-OST0000 complete [ 460.775953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 460.799068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.870594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 464.870821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.881871] Lustre: Skipped 2 previous similar messages [ 466.699977] Lustre: server umount lustre-MDT0000 complete [ 469.984484] LustreError: 16114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.014491] LustreError: 16114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 470.140243] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238776 with bad export cookie 7515054209067329330 [ 470.142060] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.155626] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 470.466850] Lustre: server umount lustre-MDT0001 complete [ 474.469280] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:32:57 (1781238777) [ 477.241262] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 479.195820] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:33:02 (1781238782) [ 485.795595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.327073] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.388133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.402511] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 491.500407] LustreError: 18990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.614680] LustreError: 18989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.147400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.537770] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 508.224093] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 511.900388] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 517.599883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 523.584649] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 528.658337] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 541.677092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.677712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.691538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 541.691545] Lustre: Skipped 1 previous similar message [ 547.566311] Lustre: server umount lustre-OST0000 complete [ 557.907326] Lustre: server umount lustre-MDT0000 complete [ 561.011853] LustreError: 18972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238866 with bad export cookie 7515054209067330212 [ 561.030380] LustreError: 18972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 561.030557] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.323356] Lustre: server umount lustre-MDT0001 complete [ 572.689354] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 576.311952] Key type lgssc unregistered [ 576.697171] LNet: 22189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.709492] LNetError: 22189:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.741136] LNet: Removed LNI 192.168.203.113@tcp [ 577.891189] Key type .llcrypt unregistered [ 577.895158] Key type ._llcrypt unregistered [ 604.117660] Key type ._llcrypt registered [ 604.122676] Key type .llcrypt registered [ 604.269246] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 605.641953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 605.784217] alg: No test for adler32 (adler32-zlib) [ 606.979884] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 607.301559] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 609.031681] Key type lgssc registered [ 610.278647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 619.437738] loop: module loaded [ 619.525172] loop0: detected capacity change from 0 to 204800000 [ 619.828987] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.852144] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.166848] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.857021] loop0: detected capacity change from 0 to 204800000 [ 626.057787] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.076548] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.094690] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 631.018392] loop0: detected capacity change from 0 to 204800000 [ 636.386840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 636.411612] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.722395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 637.779797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 637.907349] Lustre: lustre-MDT0000: new disk, initializing [ 638.040658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.068193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 642.440532] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 651.263494] loop1: detected capacity change from 0 to 204800000 [ 656.784680] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 657.029515] Lustre: lustre-OST07c6: new disk, initializing [ 657.033578] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 657.038690] Lustre: Skipped 1 previous similar message [ 657.046867] Lustre: 25342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 657.126228] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 662.803652] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 665.193120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 665.212786] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 665.266963] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 668.889249] Lustre: 24347:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 668.915796] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 668.927681] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 668.940863] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 668.951735] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 668.960230] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 683.659482] Lustre: 24346:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 683.667281] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 683.678457] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.686379] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 683.691798] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 683.699669] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 688.274460] Lustre: 24347:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 688.281515] Lustre: 24347:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 688.286175] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 688.300320] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.307294] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 688.313240] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.318860] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 688.322839] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.329347] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 688.336342] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.340837] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 688.345286] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.534387] Lustre: 24348:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 691.541690] Lustre: 24348:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 691.547922] Lustre: 24348:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 691.561131] Lustre: 24348:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.569923] Lustre: 24348:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.579489] Lustre: 24348:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.586352] Lustre: 24348:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 691.598485] Lustre: 24348:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.606513] Lustre: 24348:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 691.613445] Lustre: 24348:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.617847] Lustre: 24348:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.624404] Lustre: 24348:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.260671] Lustre: 24346:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 696.278641] Lustre: 24346:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 696.284650] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.296028] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.302516] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.307905] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.313756] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 696.320057] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.330714] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.336609] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.343158] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.360923] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.019140] Lustre: 24347:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.024372] Lustre: 24347:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 707.032923] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.037814] Lustre: 24347:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.046174] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.051727] Lustre: 24347:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.056564] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.061261] Lustre: 24347:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.065675] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.069241] Lustre: 24347:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 707.073614] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.083409] Lustre: 24347:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 757.218592] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 757.230206] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.244482] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 759.950475] Lustre: server umount lustre-OST07c6 complete [ 769.153800] Lustre: server umount lustre-MDT0000 complete [ 785.472585] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 787.984874] Key type lgssc unregistered [ 788.306673] LNet: 27862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.316198] LNetError: 27862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.339206] LNet: Removed LNI 192.168.203.113@tcp [ 789.028797] Key type .llcrypt unregistered [ 789.035048] Key type ._llcrypt unregistered [ 804.267270] Key type ._llcrypt registered [ 804.270059] Key type .llcrypt registered [ 804.385252] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 805.091186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.161141] alg: No test for adler32 (adler32-zlib) [ 806.176020] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 806.399475] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 808.071165] Key type lgssc registered [ 809.015967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 815.142280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 815.156326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.487948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.553475] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 826.385588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.719655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 829.822862] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 834.196940] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 837.201804] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 842.785484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 843.024491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.091260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 848.047880] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 852.352706] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 866.790729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.792243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.796791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 866.833433] Lustre: Skipped 1 previous similar message [ 871.569739] Lustre: server umount lustre-OST0000 complete [ 877.024663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.048421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 877.052855] Lustre: Skipped 2 previous similar messages [ 880.955264] Lustre: server umount lustre-MDT0000 complete [ 882.164384] LustreError: 29095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.199269] LustreError: 29095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 884.802310] LustreError: 29075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239190 with bad export cookie 13948122103385836652 [ 884.806235] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 884.810915] LustreError: 29075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 885.087395] Lustre: server umount lustre-MDT0001 complete [ 896.020198] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 898.896071] Key type lgssc unregistered [ 899.344166] LNet: 32474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.361090] LNetError: 32474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.382830] LNet: Removed LNI 192.168.203.113@tcp [ 900.609244] Key type .llcrypt unregistered [ 900.614793] Key type ._llcrypt unregistered [ 922.359455] Key type ._llcrypt registered [ 922.364235] Key type .llcrypt registered [ 922.501063] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 923.763475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 923.776354] alg: No test for adler32 (adler32-zlib) [ 924.968429] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 925.215247] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 926.911270] Key type lgssc registered [ 927.972607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 935.000272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 935.019613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.370473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.479343] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 946.666407] LustreError: 33737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.688070] LustreError: 33737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 948.819764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.379441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 953.791508] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 958.953971] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.344565] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 968.768441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 968.996705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.479391] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 981.173866] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 981.319792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 994.786753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.794817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.814823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.842297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.843899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 996.863611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.311713] Lustre: server umount lustre-OST0000 complete [ 1005.536281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1005.544049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.557275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.560231] Lustre: Skipped 1 previous similar message [ 1008.680162] Lustre: server umount lustre-MDT0000 complete [ 1012.041604] LustreError: 33719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239318 with bad export cookie 13908922754381215325 [ 1012.052608] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.352797] Lustre: server umount lustre-MDT0001 complete [ 1022.457972] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 1024.945821] Key type lgssc unregistered [ 1025.261550] LNet: 36913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.268341] LNetError: 36913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.278470] LNet: Removed LNI 192.168.203.113@tcp [ 1025.880525] Key type .llcrypt unregistered [ 1025.882789] Key type ._llcrypt unregistered [ 1047.310123] Key type ._llcrypt registered [ 1047.312604] Key type .llcrypt registered [ 1047.426140] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 1048.361233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.426580] alg: No test for adler32 (adler32-zlib) [ 1049.546191] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1049.901994] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 1051.591233] Key type lgssc registered [ 1052.626390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.340804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.358878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.836464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1065.950423] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1071.076961] LustreError: 38311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.096049] LustreError: 38311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1074.649607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1075.040535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.441903] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1084.793610] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1088.373116] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1095.210075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1095.616148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.451550] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1106.902200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1109.582307] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1117.156073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.163435] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.168757] Lustre: Skipped 1 previous similar message [ 1117.192313] LustreError: 38307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1117.218245] Lustre: server umount lustre-MDT0000 complete [ 1120.940857] LustreError: 38293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239426 with bad export cookie 5766891272783283075 [ 1120.958799] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.275956] LustreError: 39433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.279927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.290170] LustreError: 39433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1122.305045] Lustre: Skipped 1 previous similar message [ 1122.312209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1122.997911] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) [ 1125.280920] LustreError: 40351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.281258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1125.333692] LustreError: 40351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1127.424115] Lustre: server umount lustre-MDT0001 complete [ 1148.536141] Lustre: server umount lustre-OST0000 complete [ 1165.366673] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 1168.251388] Key type lgssc unregistered [ 1168.594189] LNet: 41815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.617685] LNetError: 41815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.655127] LNet: Removed LNI 192.168.203.113@tcp [ 1169.820073] Key type .llcrypt unregistered [ 1169.829088] Key type ._llcrypt unregistered [ 1198.018739] Key type ._llcrypt registered [ 1198.021176] Key type .llcrypt registered [ 1198.130936] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 1199.422177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1199.536239] alg: No test for adler32 (adler32-zlib) [ 1200.849715] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1201.143588] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 1202.911575] Key type lgssc registered [ 1204.081955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1212.253436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1212.288186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.785495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.731979] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1223.141781] LustreError: 43216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.162892] LustreError: 43216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1227.413031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.785901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1232.072393] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1237.899863] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1241.617752] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1247.865038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1248.230726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.258596] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1259.521214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1260.286864] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1267.680635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1267.692087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.701293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.733277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1269.737229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.740728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.770306] Lustre: Skipped 2 previous similar messages [ 1272.922662] Lustre: server umount lustre-OST0000 complete [ 1294.307898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1294.316781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.325892] Lustre: Skipped 1 previous similar message [ 1294.328365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.340137] Lustre: Skipped 1 previous similar message [ 1295.902444] Lustre: server umount lustre-MDT0000 complete [ 1299.163673] LustreError: 43197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239605 with bad export cookie 1961446573764809357 [ 1299.166051] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.178204] LustreError: 43197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1305.561457] Lustre: server umount lustre-MDT0001 complete [ 1316.038880] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 1319.177277] Key type lgssc unregistered [ 1319.588741] LNet: 46569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.611645] LNetError: 46569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.646342] LNet: Removed LNI 192.168.203.113@tcp [ 1321.171675] Key type .llcrypt unregistered [ 1321.178307] Key type ._llcrypt unregistered [ 1350.204577] Key type ._llcrypt registered [ 1350.207254] Key type .llcrypt registered [ 1350.306713] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 1351.360704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.538633] alg: No test for adler32 (adler32-zlib) [ 1352.720934] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1352.928730] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 1354.663217] Key type lgssc registered [ 1355.962907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.985893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1364.007622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.597448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1370.426773] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1375.719351] LustreError: 47969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.734982] LustreError: 47969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1379.746849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.294725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.145756] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1391.679317] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1395.483637] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1402.127168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1402.489435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.443967] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1410.673987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1414.733087] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1419.777103] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1423.438783] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1435.431198] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-12 0h48m58s [ 1436.967949] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1441.765228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1441.769738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1441.772088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1441.813995] Lustre: Skipped 2 previous similar messages [ 1442.856094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1442.870358] Lustre: Skipped 5 previous similar messages [ 1445.182208] Lustre: server umount lustre-MDT0000 complete [ 1448.103831] LustreError: 47964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.726403] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1454.256078] LustreError: 50024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.283056] LustreError: 50024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1456.240097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.341743] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.697056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.760322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1457.706852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1461.469793] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1465.227603] Lustre: DEBUG MARKER: Wait for df (36048) ... [ 1466.925314] Lustre: DEBUG MARKER: done [ 1472.480931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1472.491497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.507447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1472.510187] Lustre: Skipped 1 previous similar message [ 1478.117567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1478.122786] Lustre: Skipped 1 previous similar message [ 1483.252427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1483.263988] Lustre: Skipped 1 previous similar message [ 1486.303681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1486.523851] Lustre: server umount lustre-OST0000 complete [ 1493.480487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.485892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1493.486319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.486324] Lustre: Skipped 1 previous similar message [ 1493.489861] Lustre: Skipped 2 previous similar messages [ 1497.086163] Lustre: server umount lustre-MDT0000 complete [ 1500.642606] LustreError: 47965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.670219] LustreError: 47965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1501.296862] LustreError: 49587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239807 with bad export cookie 2166866987813622162 [ 1501.305933] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.318769] LustreError: 49587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1501.749383] Lustre: server umount lustre-MDT0001 complete [ 1513.177867] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 1515.585989] Key type lgssc unregistered [ 1515.874950] LNet: 52521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.884741] LNetError: 52521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.912743] LNet: Removed LNI 192.168.203.113@tcp [ 1516.743693] Key type .llcrypt unregistered [ 1516.751709] Key type ._llcrypt unregistered [ 1557.686697] Key type ._llcrypt registered [ 1557.693693] Key type .llcrypt registered [ 1557.830590] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 1559.167096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.277930] alg: No test for adler32 (adler32-zlib) [ 1560.455702] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1560.830869] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 1562.543232] Key type lgssc registered [ 1563.927700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1572.533544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1572.570755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.950531] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1574.008644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1574.201905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.739094] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1587.792477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.987454] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1588.055951] Lustre: 54716:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1588.390208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.853944] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1600.320306] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1604.302442] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1613.350407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1613.666028] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1613.961558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.041761] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1621.178065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1627.708371] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1635.253842] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-12 0h52m18s [ 1636.907906] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1646.207927] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1649.317837] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1649.543314] LustreError: 54723:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1669.862073] Lustre: lustre-MDT0000: Client 9b88f830-4d2d-47a5-9d6f-64071f526092 (at 192.168.203.13@tcp) reconnecting [ 1689.583861] LustreError: 54723:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1692.121768] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1694.891396] Lustre: DEBUG MARKER: done [ 1702.880489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1702.906856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.926212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1703.396027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1703.407353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.680343] Lustre: server umount lustre-OST0000 complete [ 1711.086832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1711.092316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.114806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.119673] Lustre: Skipped 2 previous similar messages [ 1712.161814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.167243] Lustre: Skipped 1 previous similar message [ 1714.209143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1714.218427] Lustre: Skipped 2 previous similar messages [ 1717.028276] Lustre: server umount lustre-MDT0000 complete [ 1720.882356] LustreError: 54708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240026 with bad export cookie 13435988186869991527 [ 1720.889563] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1720.898880] LustreError: 54708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1721.410061] Lustre: server umount lustre-MDT0001 complete [ 1733.051247] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 1735.783055] Key type lgssc unregistered [ 1736.021878] LNet: 58695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.028474] LNetError: 58695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.045309] LNet: Removed LNI 192.168.203.113@tcp [ 1736.972303] Key type .llcrypt unregistered [ 1736.974582] Key type ._llcrypt unregistered [ 1782.428728] Key type ._llcrypt registered [ 1782.430449] Key type .llcrypt registered [ 1782.555518] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 1783.634588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.753249] alg: No test for adler32 (adler32-zlib) [ 1785.004768] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1785.325452] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 1787.031389] Key type lgssc registered [ 1788.307825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1791.772191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1791.815051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.168547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1793.190578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1793.372085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.336412] Lustre: Failing over lustre-MDT0000 [ 1794.597897] Lustre: server umount lustre-MDT0000 complete [ 1801.212293] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:55:04 (1781240104) [ 1807.686651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1808.177592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.772345] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1822.588804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.737062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1822.787672] Lustre: 61476:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1823.270378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1828.133910] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1833.370244] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.846581] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1843.965487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1844.280251] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1844.534974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1851.500680] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1853.065690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1857.430880] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1862.366975] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1869.280757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1869.289573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.303275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.894554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1873.900805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1873.906747] Lustre: Skipped 1 previous similar message [ 1873.936839] Lustre: Skipped 1 previous similar message [ 1875.302614] Lustre: server umount lustre-MDT0000 complete [ 1878.914100] LustreError: 61468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240184 with bad export cookie 6277945053723087809 [ 1878.922734] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.933224] LustreError: 61468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1879.009492] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.035623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1879.041884] Lustre: Skipped 1 previous similar message [ 1879.154333] Lustre: server umount lustre-MDT0001 complete [ 1882.850660] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1887.575448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1897.252000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.524699] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x571fbdc3ae95dca4 [ 1903.546564] Lustre: MGC192.168.203.113@tcp: Connection restored to 0@lo (at 0@lo) [ 1903.923514] LustreError: 63097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.943605] LustreError: 63097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1904.042820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.138291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1908.074777] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.088930] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1908.446270] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1913.314582] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.568703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.000392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.077669] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1921.366622] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1927.339459] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1928.358930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1930.882306] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1938.912631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.923912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.927414] Lustre: Skipped 2 previous similar messages [ 1943.998203] Lustre: server umount lustre-MDT0000 complete [ 1944.036747] LustreError: 63097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.067369] LustreError: 63097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1948.057651] LustreError: 64047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240254 with bad export cookie 6277945053723090084 [ 1948.063981] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.084281] LustreError: 64047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1948.386960] Lustre: server umount lustre-MDT0001 complete [ 1954.815559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1964.196264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.535200] Lustre: 60399:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240255/real 1781240255] req@000000007d072402 x1867765609725824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1965.560658] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.580562] Lustre: Skipped 2 previous similar messages [ 1969.631736] Lustre: 60398:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240259/real 1781240259] req@00000000fc8b0461 x1867765609725952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1970.655175] Lustre: 60401:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240260/real 1781240260] req@000000000e56fff5 x1867765609726080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240276 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.791985] LustreError: 60397:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006a302b5a x1867765609726848/t0(0) o250->MGC192.168.203.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.170205] LustreError: 63098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.248395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.315265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1975.263303] Lustre: 60401:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240264/real 1781240264] req@000000005b6d9532 x1867765609726336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1979.065399] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1979.364482] Lustre: 60398:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240269/real 1781240269] req@00000000626a986d x1867765609726464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781240285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1985.508230] LustreError: 63097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.529703] LustreError: 63097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1987.341937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.610664] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 1993.214259] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1998.113682] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2002.544779] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2009.569605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.572592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.590079] Lustre: Skipped 1 previous similar message [ 2015.785533] Lustre: server umount lustre-MDT0000 complete [ 2019.507840] LustreError: 61468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240325 with bad export cookie 6277945053723091463 [ 2019.518629] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.531088] LustreError: 61468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2019.833703] LustreError: 68341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.093654] Lustre: server umount lustre-MDT0001 complete [ 2027.232253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2030.968838] Lustre: DEBUG MARKER: files compared the same [ 2033.359174] Lustre: server umount lustre-OST0000 complete [ 2050.104987] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 2053.262830] Key type lgssc unregistered [ 2053.574957] LNet: 70390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.582695] LNetError: 70390:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.633428] LNet: Removed LNI 192.168.203.113@tcp [ 2055.598897] Key type .llcrypt unregistered [ 2055.602678] Key type ._llcrypt unregistered [ 2079.079707] Key type ._llcrypt registered [ 2079.083329] Key type .llcrypt registered [ 2079.217415] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 2080.553801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.591176] alg: No test for adler32 (adler32-zlib) [ 2081.715725] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2082.037497] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 2083.746070] Key type lgssc registered [ 2084.813695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2092.470092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2092.493359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.084876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.927570] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug +malloc all [ 2104.296062] LustreError: 71791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.333056] LustreError: 71791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2106.078928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.409843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.058411] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug +malloc all [ 2116.994408] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.342120] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2126.613562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2126.940967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.585211] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug +malloc all [ 2137.870817] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2139.243836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2146.786924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.798838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.819929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.861661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.864194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.871583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.905644] Lustre: Skipped 1 previous similar message [ 2152.516724] Lustre: server umount lustre-OST0000 complete [ 2157.538899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2157.547147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2157.572071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.096189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.100307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.725026] Lustre: server umount lustre-MDT0000 complete [ 2165.215897] LustreError: 71788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.243302] LustreError: 71788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2166.893733] LustreError: 71773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240472 with bad export cookie 4177206119827303964 [ 2166.904759] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.912622] LustreError: 71773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2167.304697] Lustre: server umount lustre-MDT0001 complete [ 2178.249848] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 2180.888611] Key type lgssc unregistered [ 2181.182660] LNet: 74982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.193191] LNetError: 74982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.215181] LNet: Removed LNI 192.168.203.113@tcp [ 2182.050804] Key type .llcrypt unregistered [ 2182.054814] Key type ._llcrypt unregistered [ 2206.308239] Key type ._llcrypt registered [ 2206.310465] Key type .llcrypt registered [ 2206.402521] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 2207.378050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.498851] alg: No test for adler32 (adler32-zlib) [ 2208.844442] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2209.201093] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 2210.983147] Key type lgssc registered [ 2212.842134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2223.865567] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2223.901861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.583346] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2244.703198] LustreError: 76373:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.113@tcp: failed processing log, type 4: rc = -110 [ 2271.391378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.756958] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2283.772592] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2290.791761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.078410] Lustre: *** cfs_fail_loc=706, val=0*** [ 2291.488967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.600313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2295.884199] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2296.805298] LustreError: 76398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.847081] LustreError: 76398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2301.921969] LustreError: 76399:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.749736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.142836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2308.223579] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2313.745456] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2317.242404] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2319.331749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2319.348486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.371202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.890295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.897820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2321.918577] Lustre: Skipped 1 previous similar message [ 2325.370996] Lustre: server umount lustre-OST0000 complete [ 2329.569750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2329.583746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.606784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.131391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2332.133372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2332.168467] Lustre: Skipped 1 previous similar message [ 2336.002926] Lustre: server umount lustre-MDT0000 complete [ 2337.256944] LustreError: 77350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.271762] LustreError: 77350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2339.953870] LustreError: 76379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240645 with bad export cookie 17636981599329036429 [ 2339.967600] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.976721] LustreError: 76379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2340.427196] Lustre: server umount lustre-MDT0001 complete [ 2352.304634] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 2355.537572] Key type lgssc unregistered [ 2355.776326] LNet: 79631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.786158] LNetError: 79631:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.820955] LNet: Removed LNI 192.168.203.113@tcp [ 2356.988484] Key type .llcrypt unregistered [ 2356.998283] Key type ._llcrypt unregistered [ 2385.630922] Key type ._llcrypt registered [ 2385.635928] Key type .llcrypt registered [ 2385.722435] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 2386.752828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.863361] alg: No test for adler32 (adler32-zlib) [ 2387.957901] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2388.212378] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 2389.866521] Key type lgssc registered [ 2391.126208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2398.974329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2398.986209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.796381] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2411.833973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.417255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2416.577688] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2417.635887] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.659137] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2422.757135] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.787593] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2425.590341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2426.002847] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.019591] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2426.041371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.460475] LustreError: 81706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.485724] LustreError: 81706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2433.003774] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2436.578482] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.605674] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2441.762283] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.784376] Lustre: MGS: Client a8822bcb-b934-4c2c-bf08-483dc04a10f9 (at 0@lo) reconnecting [ 2441.801464] LustreError: 81716:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2442.088648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.168211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2446.614797] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2462.702891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2462.719543] Lustre: Skipped 1 previous similar message [ 2462.721342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2462.721516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.634560] Lustre: server umount lustre-OST0000 complete [ 2472.933066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2472.934468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.943521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2472.968254] Lustre: Skipped 1 previous similar message [ 2472.981307] Lustre: Skipped 2 previous similar messages [ 2477.672854] Lustre: server umount lustre-MDT0000 complete [ 2478.054713] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.078745] LustreError: 81701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2482.072646] Lustre: server umount lustre-MDT0001 complete [ 2485.722671] Lustre: server umount MGS complete [ 2497.524470] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:06:40 (1781240800) [ 2526.753991] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 2535.923902] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 2572.578507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.580924] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2588.006202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.252466] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2588.302558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2588.427370] Lustre: lustre-MDT0001: new disk, initializing [ 2588.501543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2588.523408] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2588.534581] Lustre: 88791:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2588.548126] LustreError: 88791:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2593.950374] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2606.073122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2606.417504] Lustre: lustre-OST0000: new disk, initializing [ 2606.423253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2606.435323] Lustre: 89540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2606.511217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2613.244390] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2617.440245] Lustre: 90003:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000041074366 x1867766241993216/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781240939 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2620.831448] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2628.605962] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.633529] Lustre: MGS: Client 7e75c3a3-7165-4d8b-a061-6a5e47cfbfee (at 0@lo) reconnecting [ 2628.698482] Lustre: 88077:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2628.720284] LustreError: 88777:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2628.744978] LustreError: 88777:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2628.777673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2628.898563] Lustre: lustre-MDT0000: new disk, initializing [ 2629.144840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.175269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2629.345446] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2630.251241] Lustre: 90499:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000088ad67d x1867766242000000/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781240952 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2634.518466] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2637.794484] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2639.351050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2639.364285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2639.588247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2665.444074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2665.456233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.469682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2670.051259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2670.058491] Lustre: Skipped 1 previous similar message [ 2671.412388] Lustre: server umount lustre-OST0000 complete [ 2680.296207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2680.299783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.313091] LustreError: Skipped 1 previous similar message [ 2680.320796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.329056] Lustre: Skipped 2 previous similar messages [ 2680.357162] Lustre: Skipped 2 previous similar messages [ 2682.245855] Lustre: server umount lustre-MDT0000 complete [ 2685.409077] LustreError: 88762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.453543] LustreError: 88762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2692.951483] Lustre: server umount lustre-MDT0001 complete [ 2696.776744] Lustre: server umount MGS complete [ 2707.633436] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:10:10 (1781241010) [ 2726.086538] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 2729.162312] Key type lgssc unregistered [ 2729.491243] LNet: 92961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.503325] LNetError: 92961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.529385] LNet: Removed LNI 192.168.203.113@tcp [ 2730.585657] Key type .llcrypt unregistered [ 2730.591197] Key type ._llcrypt unregistered [ 2749.848237] Key type ._llcrypt registered [ 2749.854831] Key type .llcrypt registered [ 2750.124471] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 2752.220737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2752.278500] alg: No test for adler32 (adler32-zlib) [ 2753.487427] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2753.884143] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 2755.686788] Key type lgssc registered [ 2757.602350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.506280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.541570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.314472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.328061] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2779.618488] LustreError: 94173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.640617] LustreError: 94173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2784.744517] LustreError: 94172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.076261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.638078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2792.002696] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2798.637933] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2803.957541] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2812.447664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2812.911315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.975167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2819.640365] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2826.005695] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2840.553320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.557761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.605993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2840.629962] Lustre: Skipped 1 previous similar message [ 2843.806803] Lustre: server umount lustre-OST0000 complete [ 2850.799947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.831259] Lustre: Skipped 1 previous similar message [ 2850.854842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.861561] Lustre: Skipped 1 previous similar message [ 2854.695766] Lustre: server umount lustre-MDT0000 complete [ 2855.906581] LustreError: 94173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.933537] LustreError: 94173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2858.466199] LustreError: 94154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241164 with bad export cookie 2212528655784070236 [ 2858.478178] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.886231] Lustre: server umount lustre-MDT0001 complete [ 2870.514066] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 2875.339581] Key type lgssc unregistered [ 2875.914576] LNet: 97350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.947404] LNetError: 97350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.974639] LNet: Removed LNI 192.168.203.113@tcp [ 2877.156513] Key type .llcrypt unregistered [ 2877.163259] Key type ._llcrypt unregistered [ 2896.838879] Key type ._llcrypt registered [ 2896.843592] Key type .llcrypt registered [ 2896.982145] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 2898.246790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2898.398277] alg: No test for adler32 (adler32-zlib) [ 2899.617954] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2899.898056] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 2901.631338] Key type lgssc registered [ 2902.923774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2910.400277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2910.410404] LustreError: 98415:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2910.415230] LustreError: 98419:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2910.434144] LustreError: 98415:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 2910.441748] LustreError: 98415:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2910.464560] LustreError: 98415:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2910.473256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2912.339186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.038350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.628828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.525778] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2931.996415] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing lsmod [ 2934.840995] LustreError: 99583:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2934.859803] LustreError: 99590:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2934.867919] LustreError: 99583:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 2934.878752] LustreError: 99583:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2934.886963] LustreError: 99583:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2934.889545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2935.306102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.415680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2941.923529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2941.930869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.944193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.433472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.448736] Lustre: Skipped 1 previous similar message [ 2942.462590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.465501] Lustre: Skipped 2 previous similar messages [ 2944.565974] Lustre: server umount lustre-MDT0000 complete [ 2946.607392] LustreError: 98452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.646231] LustreError: 98452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2949.587215] LustreError: 98433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241255 with bad export cookie 13647951161747325538 [ 2949.597369] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2949.609117] LustreError: 98433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2950.815524] LustreError: 98887:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000135762ff x1867766777795968/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2951.283189] Lustre: server umount lustre-MDT0001 complete [ 2965.938871] Lustre: server umount lustre-OST0000 complete [ 2974.828165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.327410] LustreError: 100656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.402891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.414380] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2980.836117] LustreError: 100657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.959933] LustreError: 100656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.290218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.583283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2993.813105] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 2998.734600] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3002.123937] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3009.100194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.400190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.290632] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3020.620990] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3020.668168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3029.986260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3029.997762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.018769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.012127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3035.152111] Lustre: server umount lustre-OST0000 complete [ 3040.737550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3040.746920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.753752] Lustre: Skipped 1 previous similar message [ 3040.756864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3040.761023] Lustre: Skipped 2 previous similar messages [ 3045.548880] Lustre: server umount lustre-MDT0000 complete [ 3046.378993] LustreError: 102758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.404974] LustreError: 102758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3049.855661] LustreError: 100638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241355 with bad export cookie 13647951161747326483 [ 3049.874973] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.194486] Lustre: server umount lustre-MDT0001 complete [ 3061.752131] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 3064.309375] Key type lgssc unregistered [ 3064.598809] LNet: 103835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3064.606656] LNetError: 103835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3064.629063] LNet: Removed LNI 192.168.203.113@tcp [ 3065.608566] Key type .llcrypt unregistered [ 3065.610950] Key type ._llcrypt unregistered [ 3094.850844] Key type ._llcrypt registered [ 3094.855659] Key type .llcrypt registered [ 3094.985355] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 3096.201437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3096.367141] alg: No test for adler32 (adler32-zlib) [ 3097.559432] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3097.807548] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 3099.527275] Key type lgssc registered [ 3100.772896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3108.400668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3108.429742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.920353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.262423] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3119.075072] LustreError: 105235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.089743] LustreError: 105235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3122.239304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.510600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3126.481353] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3131.128636] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3134.824599] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3142.087250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.452482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.792638] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3153.054991] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3154.742312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3163.245365] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3167.627320] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3173.344486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3173.357826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.386195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3175.406527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3175.406949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.408755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.516291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3180.522952] Lustre: Skipped 2 previous similar messages [ 3185.633258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3185.647231] Lustre: Skipped 1 previous similar message [ 3187.167358] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3187.514632] Lustre: server umount lustre-OST0000 complete [ 3194.337399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3194.349407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.363821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3197.751660] Lustre: server umount lustre-MDT0000 complete [ 3200.998386] LustreError: 105235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3201.014130] LustreError: 105235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3202.288930] LustreError: 105215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241508 with bad export cookie 7068483624267129403 [ 3202.309511] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3202.321085] LustreError: 105215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3202.851362] Lustre: server umount lustre-MDT0001 complete [ 3214.860177] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 3217.631231] Key type lgssc unregistered [ 3217.889051] LNet: 108516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3217.894066] LNetError: 108516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3217.913508] LNet: Removed LNI 192.168.203.113@tcp [ 3218.679829] Key type .llcrypt unregistered [ 3218.684522] Key type ._llcrypt unregistered [ 3233.903574] Key type ._llcrypt registered [ 3233.905877] Key type .llcrypt registered [ 3234.008654] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 3235.006066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3235.132734] alg: No test for adler32 (adler32-zlib) [ 3236.151300] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3236.354202] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 3238.015200] Key type lgssc registered [ 3239.094664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3246.285606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3246.323315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.865914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.852779] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3257.319987] LustreError: 109721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.332774] LustreError: 109721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3259.767955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.188961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3264.439785] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3269.327788] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3273.325625] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3279.853508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3280.165064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.846210] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3292.030565] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.461838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3302.880593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.894622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.905605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.908059] Lustre: Skipped 1 previous similar message [ 3302.921205] Lustre: Skipped 1 previous similar message [ 3307.361086] Lustre: server umount lustre-OST0000 complete [ 3311.584803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3311.594108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3311.615864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.122186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.123941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.628651] Lustre: server umount lustre-MDT0000 complete [ 3318.244435] LustreError: 109716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.270533] LustreError: 109716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3321.032581] LustreError: 109701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241626 with bad export cookie 9029134992524431040 [ 3321.044897] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.053515] LustreError: 109701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3321.515189] Lustre: server umount lustre-MDT0001 complete [ 3331.753584] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 3334.413775] Key type lgssc unregistered [ 3334.800822] LNet: 112899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3334.814759] LNetError: 112899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3334.849424] LNet: Removed LNI 192.168.203.113@tcp [ 3335.843595] Key type .llcrypt unregistered [ 3335.845364] Key type ._llcrypt unregistered [ 3358.304157] Key type ._llcrypt registered [ 3358.308429] Key type .llcrypt registered [ 3358.450322] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 3359.881364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3359.910202] alg: No test for adler32 (adler32-zlib) [ 3360.982544] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3361.336515] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 3363.055155] Key type lgssc registered [ 3364.214139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3372.159683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3372.205535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.669839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.895682] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3383.800194] LustreError: 114299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.810073] LustreError: 114299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3388.408398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.830049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.010946] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3400.490506] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.328489] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.258468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3411.687841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3417.532070] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3418.892502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3424.246670] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3433.141778] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3437.501611] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3437.512806] Lustre: Skipped 2 previous similar messages [ 3442.553853] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3442.559680] Lustre: Skipped 2 previous similar messages [ 3442.807293] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3442.834420] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3455.975910] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3455.989588] Lustre: Skipped 2 previous similar messages [ 3464.268378] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.13@tcp 192.168.0.[2,10]@tcp [ 3469.085474] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.13/24@tcp in log lustre-MDT0000 [ 3469.094132] Lustre: Skipped 2 previous similar messages [ 3470.841373] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.13/24@tcp [ 3470.852531] Lustre: Skipped 1 previous similar message [ 3478.000046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3478.000134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.011105] LustreError: Skipped 3 previous similar messages [ 3478.032938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.547058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.556322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.557962] LustreError: Skipped 2 previous similar messages [ 3480.557995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.589178] Lustre: Skipped 1 previous similar message [ 3482.185058] Lustre: server umount lustre-OST0000 complete [ 3485.665431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.674251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.681668] Lustre: Skipped 1 previous similar message [ 3485.694897] Lustre: Skipped 1 previous similar message [ 3490.786907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.798305] Lustre: Skipped 1 previous similar message [ 3491.960827] Lustre: server umount lustre-MDT0000 complete [ 3495.370732] LustreError: 114281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241801 with bad export cookie 11784397474590852445 [ 3495.387027] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.388119] LustreError: 114281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3495.689407] Lustre: server umount lustre-MDT0001 complete [ 3504.901560] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 3508.069300] Key type lgssc unregistered [ 3508.549436] LNet: 118780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3508.559518] LNetError: 118780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3508.583259] LNet: Removed LNI 192.168.203.113@tcp [ 3509.639791] Key type .llcrypt unregistered [ 3509.645054] Key type ._llcrypt unregistered [ 3537.284559] Key type ._llcrypt registered [ 3537.288509] Key type .llcrypt registered [ 3537.410848] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 3538.453757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.546141] alg: No test for adler32 (adler32-zlib) [ 3539.602681] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3539.830476] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 3541.520085] Key type lgssc registered [ 3542.851218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3550.151329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3550.169892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.617220] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3551.636711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3551.657473] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.13/24@tcp [ 3555.360859] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3561.958659] LustreError: 120326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.976103] LustreError: 120326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3563.140142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.370176] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3563.403209] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.13/24@tcp [ 3563.449103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.255782] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3573.659433] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3577.558615] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3584.248803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.609464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.511968] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3596.674935] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3596.918401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3607.523554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3607.528601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.531110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3607.542573] Lustre: Skipped 1 previous similar message [ 3607.555504] Lustre: Skipped 1 previous similar message [ 3612.229511] Lustre: server umount lustre-OST0000 complete [ 3617.767229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.775048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.787418] Lustre: Skipped 1 previous similar message [ 3617.800116] Lustre: Skipped 1 previous similar message [ 3622.232875] Lustre: server umount lustre-MDT0000 complete [ 3622.886749] LustreError: 120326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.909765] LustreError: 120326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3625.708744] LustreError: 120307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241931 with bad export cookie 6464851871753850555 [ 3625.714073] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.720568] LustreError: 120307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3626.161336] Lustre: server umount lustre-MDT0001 complete [ 3635.733446] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 3638.071227] Key type lgssc unregistered [ 3638.340874] LNet: 123556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3638.354308] LNetError: 123556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3638.382438] LNet: Removed LNI 192.168.203.113@tcp [ 3639.275632] Key type .llcrypt unregistered [ 3639.281334] Key type ._llcrypt unregistered [ 3664.546623] Key type ._llcrypt registered [ 3664.548498] Key type .llcrypt registered [ 3664.689365] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 3680.609265] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 3681.755874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3682.042062] alg: No test for adler32 (adler32-zlib) [ 3683.111394] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3683.523524] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 3685.407226] Key type lgssc registered [ 3686.561754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.611147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.643900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.989168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3717.034765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3717.141237] Lustre: lustre-MDT0000: new disk, initializing [ 3717.248429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.263962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3721.300466] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3730.687277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.799364] Lustre: 126922:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3730.850234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3730.853334] Lustre: Skipped 1 previous similar message [ 3730.905814] Lustre: lustre-MDT0001: new disk, initializing [ 3730.986757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3731.034561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3731.048306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3734.880540] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3742.099988] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3745.878517] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3752.282539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3752.589858] Lustre: lustre-OST0000: new disk, initializing [ 3752.592838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3752.603763] Lustre: 128662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3752.685145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.896140] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3761.719052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3761.748361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3761.829192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3767.081180] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3772.048690] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3772.378244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3777.059322] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3777.354308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3782.626275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.632056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3782.638848] LustreError: Skipped 1 previous similar message [ 3782.640940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3782.658808] Lustre: Skipped 1 previous similar message [ 3785.555781] Lustre: server umount lustre-OST0000 complete [ 3792.353300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3792.362473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.397691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3792.407724] Lustre: Skipped 1 previous similar message [ 3795.443666] Lustre: server umount lustre-MDT0000 complete [ 3799.042127] LustreError: 126915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242105 with bad export cookie 17952831331919125976 [ 3799.056439] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3799.540743] Lustre: server umount lustre-MDT0001 complete [ 3807.495965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.806292] LustreError: 130733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.822136] LustreError: 130733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3807.878436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.600049] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3813.351967] LustreError: 130734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.456617] LustreError: 130733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3819.563612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.093661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.274317] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3829.492722] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3832.967569] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3839.057446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3839.346624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3844.214357] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3849.165549] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3853.251435] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3853.467592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3858.028896] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3858.182891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3866.513060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3866.688898] Lustre: lustre-OST0001: new disk, initializing [ 3866.694543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3866.702442] Lustre: lustre-OST0001: Not available for connect from 192.168.203.13@tcp (not set up) [ 3866.710478] Lustre: 133730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3866.723611] Lustre: Skipped 1 previous similar message [ 3866.832352] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3872.902780] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 3875.909813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3875.927933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3876.026225] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3881.973665] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3882.190784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3886.272413] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3886.577638] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3890.346859] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d0c2bba800.ost_server_uuid 50 [ 3892.365049] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d0c2bba800.ost_server_uuid in FULL state after 0 sec [ 3901.924485] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.939645] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.942295] Lustre: Skipped 1 previous similar message [ 3901.948838] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3901.974619] Lustre: Skipped 1 previous similar message [ 3906.317870] Lustre: server umount lustre-OST0001 complete [ 3907.048425] LustreError: 132691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.061768] LustreError: 132691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3911.140826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3911.153726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3911.169444] Lustre: Skipped 1 previous similar message [ 3911.173907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.164969] LustreError: 132691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.181165] LustreError: 132691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3916.472483] Lustre: server umount lustre-OST0000 complete [ 3922.401514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3922.416697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.425507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.441590] Lustre: Skipped 2 previous similar messages [ 3922.453617] Lustre: Skipped 3 previous similar messages [ 3926.748786] Lustre: server umount lustre-MDT0000 complete [ 3927.521048] LustreError: 132347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.542645] LustreError: 132347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3930.240454] LustreError: 130715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242236 with bad export cookie 17952831331919126858 [ 3930.245817] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.252778] LustreError: 130715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3930.765435] Lustre: server umount lustre-MDT0001 complete [ 3949.453109] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 3953.320699] Key type lgssc unregistered [ 3953.761906] LNet: 136566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3953.784430] LNetError: 136566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3954.856425] LNet: Removed LNI 192.168.203.113@tcp [ 3956.066632] Key type .llcrypt unregistered [ 3956.070244] Key type ._llcrypt unregistered [ 3999.803915] Key type ._llcrypt registered [ 3999.806504] Key type .llcrypt registered [ 3999.939564] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 4015.449765] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 4016.642225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4016.685531] alg: No test for adler32 (adler32-zlib) [ 4017.778924] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 4018.020986] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 4019.719151] Key type lgssc registered [ 4021.018595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4059.134731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4059.153773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.353964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.382693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.427321] Lustre: lustre-MDT0000: new disk, initializing [ 4060.483185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.501485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4064.001563] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4074.492475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.705308] Lustre: 140691:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4074.770732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4074.780560] Lustre: Skipped 1 previous similar message [ 4074.862796] Lustre: lustre-MDT0001: new disk, initializing [ 4074.981653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4075.041463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4075.065761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4079.666970] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4086.612256] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4090.362902] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4096.968501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.195420] Lustre: lustre-OST0000: new disk, initializing [ 4097.198946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4097.207887] Lustre: 142432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4097.281927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.435870] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4104.279950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4104.285566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4104.420404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4110.885270] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4114.767275] Lustre: Failing over lustre-OST0000 [ 4114.782750] LustreError: 143124:0:(ldlm_resource.c:1180:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x2:0x0].0x0 (00000000b11b4401) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4114.916716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.920227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.924493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.929442] Lustre: Skipped 1 previous similar message [ 4114.951831] LustreError: Skipped 1 previous similar message [ 4117.063401] Lustre: server umount lustre-OST0000 complete [ 4135.468609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4135.810423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4135.829742] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4141.469394] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4143.257708] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4145.263164] Lustre: Failing over lustre-MDT0000 [ 4145.289930] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.293833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4145.308378] Lustre: Skipped 2 previous similar messages [ 4145.324808] Lustre: Skipped 3 previous similar messages [ 4145.548700] Lustre: server umount lustre-MDT0000 complete [ 4153.508810] LustreError: 141931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.539358] LustreError: 141931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4155.880344] LustreError: 141930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.899157] LustreError: 141930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4158.629181] LustreError: 140699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.996176] LustreError: 140704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.042262] LustreError: 140704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4164.134382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.261258] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4164.445823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4164.486954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4164.544808] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4164.818814] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4168.754745] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4169.737048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4169.776060] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4169.828880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4179.943946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.950032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4179.966536] Lustre: Skipped 1 previous similar message [ 4179.991492] Lustre: Skipped 1 previous similar message [ 4185.062305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4185.075743] Lustre: Skipped 1 previous similar message [ 4190.178322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.199482] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4191.371372] Lustre: server umount lustre-OST0000 complete [ 4195.297709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.299572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4200.420945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4200.426697] Lustre: Skipped 4 previous similar messages [ 4200.855380] Lustre: server umount lustre-MDT0000 complete [ 4204.079053] LustreError: 140684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242510 with bad export cookie 8675129902941897023 [ 4204.081849] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.090244] LustreError: 140684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4204.480524] Lustre: server umount lustre-MDT0001 complete [ 4214.203879] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 4216.917953] Key type lgssc unregistered [ 4217.190688] LNet: 145921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4217.195192] LNetError: 145921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4218.214388] LNet: Removed LNI 192.168.203.113@tcp [ 4219.030688] Key type .llcrypt unregistered [ 4219.033251] Key type ._llcrypt unregistered [ 4252.728071] Key type ._llcrypt registered [ 4252.729653] Key type .llcrypt registered [ 4252.858975] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 4254.019486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4254.051630] alg: No test for adler32 (adler32-zlib) [ 4255.035609] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 4255.220161] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 4256.864711] Key type lgssc registered [ 4257.865399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4264.497925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4264.530260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.933135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.587724] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4275.173133] LustreError: 147425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.201832] LustreError: 147425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4276.211802] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4279.869742] LustreError: 147454:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4279.879706] LustreError: 147454:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4279.898476] LustreError: 147454:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4280.404530] Lustre: server umount lustre-MDT0000 complete [ 4294.353780] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 4304.727990] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 4339.755542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.969555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4340.029983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4340.138615] Lustre: lustre-MDT0000: new disk, initializing [ 4340.233810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.247993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4343.773805] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4353.620563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.713725] Lustre: 150612:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4353.771749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4353.777548] Lustre: Skipped 1 previous similar message [ 4353.843233] Lustre: lustre-MDT0001: new disk, initializing [ 4353.945147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4353.987814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4354.002705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4358.207136] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4365.025433] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4368.174881] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4373.959441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.171517] Lustre: lustre-OST0000: new disk, initializing [ 4374.175290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4374.181314] Lustre: 152356:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4374.259872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.698878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4379.704151] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4379.707904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4379.829730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4386.309427] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4390.071657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4392.019617] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:38:15 (1781242695) [ 4615.652718] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.666256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.672959] Lustre: Skipped 1 previous similar message [ 4620.255745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4620.260394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.283713] Lustre: Skipped 1 previous similar message [ 4620.292884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.741054] Lustre: server umount lustre-MDT0000 complete [ 4624.757705] LustreError: 150606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242930 with bad export cookie 11650217241972827136 [ 4624.758488] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.778507] LustreError: 150606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4625.892067] LustreError: 150619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.893461] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.939888] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4625.945470] Lustre: Skipped 2 previous similar messages [ 4629.919646] LustreError: 150621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4629.919763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4635.043687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4635.048274] LustreError: 150619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4639.199115] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4639.407425] Lustre: server umount lustre-MDT0001 complete [ 4648.858733] Lustre: server umount lustre-OST0000 complete [ 4656.811333] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 4666.212492] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 4696.481077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.671463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4696.718214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4696.794215] Lustre: lustre-MDT0000: new disk, initializing [ 4696.869897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4696.888564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4700.643923] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4710.246861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.374906] Lustre: 156874:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4710.419468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4710.424301] Lustre: Skipped 1 previous similar message [ 4710.493839] Lustre: lustre-MDT0001: new disk, initializing [ 4710.625390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4710.682044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4710.697872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4714.709978] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4721.497336] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4724.602195] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4730.111101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4730.292311] Lustre: lustre-OST0000: new disk, initializing [ 4730.295923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4730.301055] Lustre: 158614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4730.378626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.174532] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4738.620686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4738.626654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4738.745680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4741.887674] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4745.676660] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4745.855786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4749.763974] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4749.964514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4754.403118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4754.403850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.409224] LustreError: Skipped 1 previous similar message [ 4754.418661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.775824] Lustre: server umount lustre-OST0000 complete [ 4762.085609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4762.102076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.116994] Lustre: Skipped 1 previous similar message [ 4767.438347] Lustre: server umount lustre-MDT0000 complete [ 4769.760214] LustreError: 156886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.770158] LustreError: 156886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4770.094419] LustreError: 156865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243076 with bad export cookie 11650217241972985966 [ 4770.095514] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4770.105556] LustreError: 156865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4770.403402] Lustre: server umount lustre-MDT0001 complete [ 4777.113626] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:44:40 (1781243080) [ 4788.470577] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 4796.462592] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 4823.038461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.282457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4823.311858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4823.392517] Lustre: lustre-MDT0000: new disk, initializing [ 4823.459812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.486712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4826.470392] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4834.648388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.760578] Lustre: 163188:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.008340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4838.546727] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4844.266960] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4847.018991] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4852.780406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4853.062651] Lustre: 164934:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4857.684674] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4863.025689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4863.030609] Lustre: Skipped 1 previous similar message [ 4863.033568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4863.112633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4863.353841] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4873.192455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4873.193909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4873.203252] Lustre: Skipped 2 previous similar messages [ 4873.208251] Lustre: Skipped 4 previous similar messages [ 4873.218885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4875.166160] Lustre: server umount lustre-OST0000 complete [ 4883.979088] Lustre: server umount lustre-MDT0000 complete [ 4886.791229] LustreError: 163180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243192 with bad export cookie 11650217241972986848 [ 4886.794243] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4886.804985] LustreError: 163180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4887.086743] Lustre: server umount lustre-MDT0001 complete [ 4894.207710] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:46:37 (1781243197) [ 4906.948598] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 4914.824176] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 4944.154755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.420985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4944.452086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4944.457602] Lustre: Skipped 3 previous similar messages [ 4944.527280] Lustre: lustre-MDT0000: new disk, initializing [ 4944.530799] Lustre: Skipped 2 previous similar messages [ 4944.613246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.623232] Lustre: Skipped 2 previous similar messages [ 4944.639950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4947.936840] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4956.447193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.658933] Lustre: 169108:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4956.920944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4960.506737] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4965.992531] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4968.957261] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4974.247286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.499576] Lustre: 170861:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4979.153938] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 4981.930741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4985.437666] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4991.971617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4991.977919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4991.988689] Lustre: Skipped 2 previous similar messages [ 4991.998138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4992.013972] Lustre: Skipped 5 previous similar messages [ 4997.099521] Lustre: server umount lustre-OST0000 complete [ 5006.367130] Lustre: server umount lustre-MDT0000 complete [ 5007.848113] LustreError: 170504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.872660] LustreError: 170504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5009.422696] LustreError: 170860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243315 with bad export cookie 11650217241972987933 [ 5009.438426] LustreError: 170860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5009.439337] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5009.724992] Lustre: server umount lustre-MDT0001 complete [ 5018.034873] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 5020.160094] Key type lgssc unregistered [ 5020.393190] LNet: 172585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5020.401021] LNetError: 172585:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5020.424446] LNet: Removed LNI 192.168.203.113@tcp [ 5021.082847] Key type .llcrypt unregistered [ 5021.085670] Key type ._llcrypt unregistered [ 5038.289658] Key type ._llcrypt registered [ 5038.293028] Key type .llcrypt registered [ 5038.366503] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 5039.091472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5039.156086] alg: No test for adler32 (adler32-zlib) [ 5040.171927] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5040.375154] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 5042.031354] Key type lgssc registered [ 5042.932290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5049.708891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5049.729252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.140917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.653797] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5061.492035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.600085] LustreError: 173964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.612318] LustreError: 173964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5061.835525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5065.565444] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5070.945784] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5074.113977] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5079.399947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5079.621408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.952542] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5087.740153] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5097.069196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5112.806247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.811161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.817708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5112.824562] Lustre: Skipped 1 previous similar message [ 5117.921689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5117.929547] Lustre: Skipped 2 previous similar messages [ 5118.724340] Lustre: server umount lustre-OST0000 complete [ 5123.040494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.054107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.061445] Lustre: Skipped 1 previous similar message [ 5123.070442] Lustre: Skipped 1 previous similar message [ 5127.598690] Lustre: server umount lustre-MDT0000 complete [ 5128.160495] LustreError: 175012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5128.179554] LustreError: 175012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5130.244309] LustreError: 173943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243436 with bad export cookie 400752089753967787 [ 5130.247916] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.261459] LustreError: 173943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5130.505625] Lustre: server umount lustre-MDT0001 complete [ 5137.754334] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 5139.482383] Key type lgssc unregistered [ 5139.692699] LNet: 177157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5139.699500] LNetError: 177157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5139.717537] LNet: Removed LNI 192.168.203.113@tcp [ 5140.284532] Key type .llcrypt unregistered [ 5140.287216] Key type ._llcrypt unregistered [ 5155.212234] Key type ._llcrypt registered [ 5155.214414] Key type .llcrypt registered [ 5155.287844] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 5155.949602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5156.006564] alg: No test for adler32 (adler32-zlib) [ 5156.963351] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5157.146318] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 5158.783199] Key type lgssc registered [ 5159.641747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5164.714412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5164.730855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.029408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.861936] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5174.971932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.234877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5177.721612] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5181.150361] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5183.440549] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5187.564717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.838216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.605366] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5191.916093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5195.284638] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5205.475810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5205.477219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.481141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5205.496884] Lustre: Skipped 1 previous similar message [ 5206.407328] Lustre: server umount lustre-OST0000 complete [ 5210.139638] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89d0c8676000.ost_server_uuid 50 [ 5226.017724] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5241.314089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.328588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5241.333629] Lustre: Skipped 2 previous similar messages [ 5244.212255] Lustre: server umount lustre-MDT0000 complete [ 5245.408654] LustreError: 179246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.428071] LustreError: 179246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5246.828327] LustreError: 180150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243552 with bad export cookie 4260305756932906170 [ 5246.830429] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5246.835649] LustreError: 180150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5247.060204] Lustre: server umount lustre-MDT0001 complete [ 5252.122364] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:52:36 (1781243556) [ 5256.457482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5256.761699] LustreError: 182033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5256.809363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.447323] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5262.309154] LustreError: 182034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.081728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.342689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5267.996235] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5271.668703] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5274.172046] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5278.774260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5278.997346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.072512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5282.723206] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5286.422446] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5290.810140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.940701] Lustre: lustre-OST0001: new disk, initializing [ 5290.944453] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5290.949926] Lustre: 184534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5294.687377] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5298.731629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5298.741551] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5298.805044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5299.973341] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5303.781625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.783823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5303.787468] Lustre: Skipped 2 previous similar messages [ 5303.793363] Lustre: Skipped 1 previous similar message [ 5307.861879] Lustre: server umount lustre-OST0000 complete [ 5308.900599] LustreError: 184477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.911267] LustreError: 184477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5311.622721] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5311.752610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5314.023244] LustreError: 183627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5314.034775] LustreError: 183627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5314.192727] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5314.349418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5324.264453] LustreError: 184477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.277893] LustreError: 184477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5329.378374] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.381265] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5329.392703] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5329.401032] LustreError: Skipped 1 previous similar message [ 5329.404970] Lustre: Skipped 2 previous similar messages [ 5332.226270] Lustre: server umount lustre-OST0001 complete [ 5334.497429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.504133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.506771] Lustre: Skipped 2 previous similar messages [ 5334.517357] Lustre: Skipped 1 previous similar message [ 5340.631537] Lustre: server umount lustre-MDT0000 complete [ 5342.779757] LustreError: 182013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243648 with bad export cookie 4260305756932907227 [ 5342.781670] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.785070] LustreError: 182013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5342.958846] Lustre: server umount lustre-MDT0001 complete [ 5356.671688] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:54:20 (1781243660) [ 5361.011158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.211822] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5361.246224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5361.422275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.426526] Lustre: Skipped 1 previous similar message [ 5363.994435] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5369.218980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.302727] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5369.322443] Lustre: 187821:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5372.114496] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5375.582283] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.813153] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5381.807315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5381.931612] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5383.330415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5385.737901] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5389.114149] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5393.218942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.271741] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5396.524643] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5398.513561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5399.644924] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5403.625163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.640417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5403.650389] Lustre: Skipped 3 previous similar messages [ 5407.322057] Lustre: server umount lustre-OST0000 complete [ 5408.743378] LustreError: 189444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.756142] LustreError: 189444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5424.102924] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.104429] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.114734] LustreError: Skipped 1 previous similar message [ 5424.122643] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5424.122656] Lustre: Skipped 1 previous similar message [ 5424.182806] Lustre: Skipped 2 previous similar messages [ 5429.753983] Lustre: server umount lustre-OST0001 complete [ 5436.384231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5449.186291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5449.477589] Lustre: server umount lustre-MDT0000 complete [ 5449.696554] LustreError: 187833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.720188] LustreError: 187833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5454.250970] LustreError: 187814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243760 with bad export cookie 4260305756932908627 [ 5454.254875] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.265676] LustreError: 187814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5455.116890] Lustre: server umount lustre-MDT0001 complete [ 5481.320957] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:56:24 (1781243784) [ 5498.362830] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 5507.980872] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 5539.566218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.828437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5539.861978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5539.959183] Lustre: lustre-MDT0000: new disk, initializing [ 5540.039722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.046734] Lustre: Skipped 3 previous similar messages [ 5540.061242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5543.785125] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5552.858291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.979207] Lustre: 195424:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5553.018998] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5553.023202] Lustre: Skipped 1 previous similar message [ 5553.108918] Lustre: lustre-MDT0001: new disk, initializing [ 5553.226822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5553.235216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5557.340986] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5564.172327] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5567.693356] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5573.455111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5573.674775] Lustre: lustre-OST0000: new disk, initializing [ 5573.679260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5573.685387] Lustre: 197163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5578.971488] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5583.458790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5583.473722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5583.581349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5585.776255] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5590.013375] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5590.157401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5594.087973] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5594.291682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5599.206840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5599.208085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.216712] LustreError: Skipped 1 previous similar message [ 5599.221119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5599.221126] Lustre: Skipped 8 previous similar messages [ 5599.248566] Lustre: Skipped 3 previous similar messages [ 5602.222681] Lustre: server umount lustre-OST0000 complete [ 5609.441109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5612.009790] Lustre: server umount lustre-MDT0000 complete [ 5614.568533] LustreError: 197172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.586499] LustreError: 197172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5615.314965] LustreError: 195416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243921 with bad export cookie 4260305756932910412 [ 5615.333059] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.688897] Lustre: server umount lustre-MDT0001 complete [ 5623.173936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.618484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5623.624585] Lustre: Skipped 2 previous similar messages [ 5627.799548] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5635.408615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.579462] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5644.994713] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5648.549392] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5654.906841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5660.441719] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5665.794291] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5670.121336] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5670.405955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5674.897459] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5675.166166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5679.094920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5679.101358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.104967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.126351] Lustre: Skipped 3 previous similar messages [ 5679.139309] Lustre: Skipped 4 previous similar messages [ 5691.359179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5691.654429] Lustre: server umount lustre-OST0000 complete [ 5698.208647] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5703.476393] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5733.797628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.899701] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5744.424106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.436937] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.437478] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5746.645582] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5750.345917] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 5754.855924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5754.866307] LustreError: Skipped 1 previous similar message [ 5760.590259] Lustre: server umount lustre-OST0000 complete [ 5770.679234] Lustre: server umount lustre-MDT0000 complete [ 5774.952133] LustreError: 199216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244080 with bad export cookie 4260305756932911294 [ 5774.955585] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5774.961561] LustreError: 199216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5775.248864] Lustre: server umount lustre-MDT0001 complete [ 5784.863655] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:01:27 (1781244087) [ 5792.405647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.843049] LustreError: 204692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.865970] LustreError: 204692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5792.939749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.949451] Lustre: Skipped 3 previous similar messages [ 5797.971963] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5807.528363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.714450] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5819.932549] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5824.017361] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5831.184974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.495193] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5844.903735] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5850.398889] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5850.849057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5856.786650] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5857.035378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5864.227555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5864.490906] Lustre: lustre-OST0001: new disk, initializing [ 5864.494577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5864.504348] Lustre: 207690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5870.471889] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5873.755883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5873.773700] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5873.859116] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5877.779236] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5882.286466] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5882.486224] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5886.895753] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5887.140931] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5889.516643] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.517915] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5889.519056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5889.533611] Lustre: Skipped 5 previous similar messages [ 5889.555191] Lustre: Skipped 13 previous similar messages [ 5903.327602] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5903.478770] Lustre: server umount lustre-OST0001 complete [ 5909.437849] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5909.602933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5913.726372] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5913.945937] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5943.674290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5949.458545] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5949.476570] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5949.477232] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5949.508767] Lustre: Skipped 2 previous similar messages [ 5950.898750] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 5956.830610] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5958.572242] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5966.754589] Lustre: server umount lustre-OST0001 complete [ 5999.075496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5999.262982] Lustre: server umount lustre-OST0000 complete [ 6009.801774] Lustre: server umount lustre-MDT0000 complete [ 6013.877938] LustreError: 204673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244319 with bad export cookie 4260305756932912253 [ 6013.881820] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.904078] LustreError: 204673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6020.679186] Lustre: server umount lustre-MDT0001 complete [ 6048.050710] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:05:51 (1781244351) [ 6055.798129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.165170] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6056.211305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6056.457908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.466217] Lustre: Skipped 4 previous similar messages [ 6061.631399] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6071.740465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.880235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6071.907517] Lustre: 212644:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6077.414481] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6083.665593] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6087.857499] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6096.047110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.447810] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6104.064721] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6110.605607] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6120.654102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.748838] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6127.100378] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6133.015505] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6134.608870] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6139.105146] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6139.356460] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6143.774505] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6144.014691] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6147.804604] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d0d051f800.ost_server_uuid 50 [ 6149.179187] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89d0d051f800.ost_server_uuid in FULL state after 0 sec [ 6150.316456] Lustre: Permanently deactivating lustre-OST0001 [ 6150.330283] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6154.525560] Lustre: Permanently reactivating lustre-OST0001 [ 6154.544425] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6154.547933] Lustre: Skipped 2 previous similar messages [ 6159.843893] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.846253] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6159.857048] Lustre: Skipped 7 previous similar messages [ 6159.873777] Lustre: Skipped 15 previous similar messages [ 6165.322232] Lustre: server umount lustre-OST0001 complete [ 6170.085358] LustreError: 214261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.113369] LustreError: 214261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6183.391694] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6183.644199] Lustre: server umount lustre-OST0000 complete [ 6190.065519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6190.080874] LustreError: Skipped 2 previous similar messages [ 6195.090723] Lustre: server umount lustre-MDT0000 complete [ 6200.226188] LustreError: 212636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244506 with bad export cookie 4260305756932913548 [ 6200.235291] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6200.240584] LustreError: 212636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6200.804751] Lustre: server umount lustre-MDT0001 complete [ 6229.190486] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:08:52 (1781244532) [ 6237.513344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.800377] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6237.825552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6237.832570] Lustre: Skipped 2 previous similar messages [ 6242.315925] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6251.752915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.826892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6251.879215] Lustre: 218756:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6256.499097] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6263.911174] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6269.456531] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6278.783342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.942942] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6278.978617] Lustre: Permanently deactivating lustre-OST0000 [ 6278.995847] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6284.843848] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6284.864785] LustreError: 220682:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 6284.885154] LustreError: 220682:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 6285.989869] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6292.966714] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6300.945311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6301.038614] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6307.087629] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6311.167888] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6312.338656] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6314.371327] Lustre: Permanently reactivating lustre-OST0000 [ 6314.393542] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6314.402082] Lustre: Skipped 2 previous similar messages [ 6316.418972] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6316.421051] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000000992192f, cur 1781244622 deadline 1781244620 last 1781244590 [ 6316.426170] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.427625] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6316.508193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6329.661133] Lustre: server umount lustre-OST0001 complete [ 6339.823667] Lustre: server umount lustre-OST0000 complete [ 6351.125465] Lustre: server umount lustre-MDT0000 complete [ 6355.289455] LustreError: 218747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244661 with bad export cookie 4260305756932915193 [ 6355.295923] LustreError: 218747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6355.303948] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.931184] Lustre: server umount lustre-MDT0001 complete [ 6367.340931] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 6370.338644] Key type lgssc unregistered [ 6370.648078] LNet: 223165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6370.660385] LNetError: 223165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6370.687380] LNet: Removed LNI 192.168.203.113@tcp [ 6371.537876] Key type .llcrypt unregistered [ 6371.539867] Key type ._llcrypt unregistered [ 6395.092750] Key type ._llcrypt registered [ 6395.095884] Key type .llcrypt registered [ 6395.218048] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 6396.607372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6396.625329] alg: No test for adler32 (adler32-zlib) [ 6397.702905] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6397.936512] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 6399.663131] Key type lgssc registered [ 6400.902099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6411.830398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6411.852116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6413.572500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6418.231620] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6424.035963] LustreError: 224617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6424.065541] LustreError: 224617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6427.058556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.228889] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6427.306971] Lustre: 224604:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6427.328706] Lustre: Permanently deactivating lustre-MDT0001 [ 6427.372358] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6427.849975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6429.142907] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6429.155658] LustreError: 224648:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6429.179426] LustreError: 224648:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6429.192512] LustreError: 224648:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6432.895115] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6438.989394] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6444.304020] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6452.949243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.524775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.566053] LustreError: 226230:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.574391] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6458.676715] LustreError: 226231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6458.694332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6458.708681] LustreError: 226231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6461.104416] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6463.974514] LustreError: 226452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.994569] LustreError: 226452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6467.627454] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6469.098888] LustreError: 226231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6469.140663] LustreError: 226231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6475.390630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6475.704392] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6480.886676] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6480.889979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6481.810332] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6487.009491] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6489.668371] Lustre: Permanently reactivating lustre-MDT0001 [ 6489.681880] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6489.688962] Lustre: Skipped 2 previous similar messages [ 6492.616703] Lustre: lustre-MDT0001: Client c6d861e5-8d6c-4e62-b0e5-eb7a592fc22b (at 192.168.203.13@tcp) reconnecting [ 6497.705463] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6499.743750] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6506.454434] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6506.641524] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6512.818665] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6513.233919] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6515.877087] Lustre: Permanently deactivating lustre-MDT0001 [ 6515.919668] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6515.929432] Lustre: Skipped 2 previous similar messages [ 6526.071333] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6535.140884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6535.146965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.172156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.208419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.216281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.220176] Lustre: Skipped 1 previous similar message [ 6538.245248] Lustre: Skipped 2 previous similar messages [ 6541.469809] Lustre: server umount lustre-MDT0000 complete [ 6543.330782] LustreError: 224617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.355342] LustreError: 224617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6545.260529] LustreError: 224596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244851 with bad export cookie 12761084903946104805 [ 6545.269788] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.280018] LustreError: 224596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6545.685193] Lustre: server umount lustre-MDT0001 complete [ 6551.525393] Lustre: server umount lustre-OST0000 complete [ 6562.031404] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:14:25 (1781244865) [ 6572.337507] Lustre: server umount lustre-OST0001 complete [ 6579.293285] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 6588.608983] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 6620.907441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.210156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6621.221589] Lustre: Skipped 2 previous similar messages [ 6621.248757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6621.351499] Lustre: lustre-MDT0000: new disk, initializing [ 6621.493098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.511689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6625.850211] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6636.566759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.694365] Lustre: 232308:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6636.784102] LustreError: 232319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.808774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6636.813766] Lustre: Skipped 1 previous similar message [ 6636.907920] Lustre: lustre-MDT0001: new disk, initializing [ 6637.003307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6637.098992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6637.109453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6641.045709] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6648.179919] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6651.638860] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6658.492055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.895225] Lustre: lustre-OST0000: new disk, initializing [ 6658.909733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6658.923777] Lustre: 234048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6659.051329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.983592] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6668.371114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6668.382731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6668.488902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6672.520957] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6674.408136] LustreError: 232314:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6682.640382] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.791398] Lustre: lustre-OST0001: new disk, initializing [ 6682.795910] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6682.802838] Lustre: 235048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6684.495140] LustreError: 232314:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6684.517916] LustreError: 232316:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6689.394346] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6690.909362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6690.920226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6691.025946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6694.559143] LustreError: 232316:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6697.748131] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6699.748270] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6703.313450] Lustre: lustre-OST0001: Not available for connect from 192.168.203.13@tcp (stopping) [ 6703.333825] Lustre: Skipped 2 previous similar messages [ 6706.665676] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6706.669780] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.672976] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6706.688017] LustreError: Skipped 1 previous similar message [ 6706.729168] Lustre: Skipped 1 previous similar message [ 6707.954118] Lustre: server umount lustre-OST0001 complete [ 6708.391096] LustreError: 234031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.088252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.089856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.093728] LustreError: Skipped 1 previous similar message [ 6725.096839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.096846] Lustre: Skipped 1 previous similar message [ 6725.123353] Lustre: Skipped 1 previous similar message [ 6730.815957] Lustre: server umount lustre-OST0000 complete [ 6734.307690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6734.318407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.345647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6734.347753] Lustre: Skipped 3 previous similar messages [ 6740.449575] Lustre: server umount lustre-MDT0000 complete [ 6743.935424] LustreError: 232301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245049 with bad export cookie 12761084903946107619 [ 6743.947460] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6743.954465] LustreError: 232301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6758.369724] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6758.726712] Lustre: server umount lustre-MDT0001 complete [ 6768.641344] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 6771.454920] Key type lgssc unregistered [ 6771.776694] LNet: 236954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6771.787649] LNetError: 236954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6771.815912] LNet: Removed LNI 192.168.203.113@tcp [ 6772.529668] Key type .llcrypt unregistered [ 6772.532552] Key type ._llcrypt unregistered [ 6809.965676] Key type ._llcrypt registered [ 6809.968107] Key type .llcrypt registered [ 6810.055858] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 6810.948138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6810.965242] alg: No test for adler32 (adler32-zlib) [ 6812.142919] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6812.341707] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 6814.047400] Key type lgssc registered [ 6815.351969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6823.227711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6823.251950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6824.594778] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6824.615623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6824.764045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6828.804262] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6837.082051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.212352] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6837.263423] Lustre: 239146:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6837.563213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6841.943822] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6847.809972] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6851.290476] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6857.564426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6857.735879] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6857.935088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.749354] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6867.156649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6869.272695] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6923.746480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.747371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.761413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6923.783242] Lustre: Skipped 1 previous similar message [ 6928.870963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6928.878638] Lustre: Skipped 2 previous similar messages [ 6929.926460] Lustre: server umount lustre-OST0000 complete [ 6934.404369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6945.916134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6946.174927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.683244] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 6953.318290] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6953.338403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6953.341175] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6953.369043] Lustre: Skipped 1 previous similar message [ 6955.116394] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6994.401660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.412072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.451060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6994.458031] LustreError: Skipped 1 previous similar message [ 6999.530618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6999.542396] Lustre: Skipped 1 previous similar message [ 7000.247692] Lustre: server umount lustre-OST0000 complete [ 7006.692074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7006.720732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.764395] Lustre: Skipped 1 previous similar message [ 7006.784287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7006.793953] Lustre: Skipped 1 previous similar message [ 7009.765157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.708416] Lustre: server umount lustre-MDT0000 complete [ 7017.247445] LustreError: 239138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245323 with bad export cookie 17617775291043824330 [ 7017.250212] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.267843] LustreError: 239138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7017.834792] Lustre: server umount lustre-MDT0001 complete [ 7029.805600] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 7032.611274] Key type lgssc unregistered [ 7032.974957] LNet: 244104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7032.984584] LNetError: 244104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7032.997884] LNet: Removed LNI 192.168.203.113@tcp [ 7033.832868] Key type .llcrypt unregistered [ 7033.835601] Key type ._llcrypt unregistered [ 7056.210829] Key type ._llcrypt registered [ 7056.213547] Key type .llcrypt registered [ 7056.346582] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 7057.181850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7057.351606] alg: No test for adler32 (adler32-zlib) [ 7058.485192] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7058.752273] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 7060.399165] Key type lgssc registered [ 7061.952918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7071.580718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7071.630806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7073.082093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.767919] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7083.490413] LustreError: 245504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7083.531907] LustreError: 245504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7088.330279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.615407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7088.767448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7092.968040] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7098.549524] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7102.772827] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7109.379722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7109.647845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.187516] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7118.901806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7121.223782] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7144.416924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7144.422040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.431634] LustreError: Skipped 1 previous similar message [ 7144.433953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7144.456116] Lustre: Skipped 1 previous similar message [ 7144.475506] Lustre: Skipped 1 previous similar message [ 7148.478682] Lustre: server umount lustre-OST0000 complete [ 7154.662686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.666069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7154.681160] Lustre: Skipped 1 previous similar message [ 7154.698431] Lustre: Skipped 1 previous similar message [ 7158.211147] Lustre: server umount lustre-MDT0000 complete [ 7159.783038] LustreError: 245505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.802314] LustreError: 245505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7161.438571] LustreError: 245484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245467 with bad export cookie 13828545217003872268 [ 7161.445823] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7161.707680] Lustre: server umount lustre-MDT0001 complete [ 7170.084060] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 7172.500816] Key type lgssc unregistered [ 7172.771718] LNet: 249456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7172.776267] LNetError: 249456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7172.787946] LNet: Removed LNI 192.168.203.113@tcp [ 7173.587566] Key type .llcrypt unregistered [ 7173.591336] Key type ._llcrypt unregistered [ 7196.566584] Key type ._llcrypt registered [ 7196.570727] Key type .llcrypt registered [ 7196.752942] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 7198.126995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7198.320077] alg: No test for adler32 (adler32-zlib) [ 7199.744712] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7200.478977] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 7202.319597] Key type lgssc registered [ 7204.451348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7212.625226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7212.642701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.085519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7218.205454] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7224.306450] LustreError: 250661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.332858] LustreError: 250661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7226.340043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.730452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7230.586543] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7236.233843] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7241.170697] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7249.771611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.118487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7257.915327] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7259.518681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7263.516793] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7274.980214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.983188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7274.996364] Lustre: Skipped 1 previous similar message [ 7279.485811] Lustre: server umount lustre-OST0000 complete [ 7283.169230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7283.183334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.200735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.203822] Lustre: Skipped 1 previous similar message [ 7285.225135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.225388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.249909] Lustre: Skipped 1 previous similar message [ 7288.923629] Lustre: server umount lustre-MDT0000 complete [ 7290.338000] LustreError: 250656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7290.371339] LustreError: 250656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7292.212636] LustreError: 251377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245598 with bad export cookie 14650613059715391785 [ 7292.217149] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.221818] LustreError: 251377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7292.501785] Lustre: server umount lustre-MDT0001 complete [ 7302.476576] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 7304.621878] Key type lgssc unregistered [ 7304.875582] LNet: 253979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7304.883326] LNetError: 253979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7304.901902] LNet: Removed LNI 192.168.203.113@tcp [ 7305.610918] Key type .llcrypt unregistered [ 7305.613087] Key type ._llcrypt unregistered [ 7323.817914] Key type ._llcrypt registered [ 7323.819891] Key type .llcrypt registered [ 7323.910094] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 7324.710511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7324.736224] alg: No test for adler32 (adler32-zlib) [ 7325.719957] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7325.916719] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 7327.591306] Key type lgssc registered [ 7328.485791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7334.966312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7334.985632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.342193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.530767] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7346.157290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7346.495595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.603106] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7354.055406] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7357.190423] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7362.930556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7363.206403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.703736] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7371.619207] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7382.702856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7402.465859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7402.475449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.495735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7403.492424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7403.493762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7403.495537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.518263] Lustre: Skipped 1 previous similar message [ 7406.541270] Lustre: server umount lustre-OST0000 complete [ 7413.216364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7413.223209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.235387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.867495] Lustre: server umount lustre-MDT0000 complete [ 7418.857516] LustreError: 255354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.876928] LustreError: 255354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7419.267496] LustreError: 255339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245725 with bad export cookie 3392249202917723458 [ 7419.268778] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.286026] LustreError: 255339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7420.052774] Lustre: server umount lustre-MDT0001 complete [ 7436.615975] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 7440.846144] Key type lgssc unregistered [ 7441.206597] LNet: 259329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7441.214280] LNetError: 259329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7441.232058] LNet: Removed LNI 192.168.203.113@tcp [ 7442.806119] Key type .llcrypt unregistered [ 7442.816425] Key type ._llcrypt unregistered [ 7464.972995] Key type ._llcrypt registered [ 7464.980383] Key type .llcrypt registered [ 7465.102461] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 7466.734221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7466.759071] alg: No test for adler32 (adler32-zlib) [ 7467.869797] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7468.232030] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 7469.968267] Key type lgssc registered [ 7471.677543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7480.107760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7480.140627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.746783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.668347] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7492.071313] LustreError: 260550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.112257] LustreError: 260550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7497.192564] LustreError: 260552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.407355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.897311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7502.050086] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7508.547204] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7512.293855] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7519.936285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7520.335832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.676239] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7527.564058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7533.745346] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7548.395916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7548.406801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.411378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7548.450988] Lustre: Skipped 1 previous similar message [ 7551.939394] Lustre: server umount lustre-OST0000 complete [ 7558.634173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.654082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7558.671824] Lustre: Skipped 1 previous similar message [ 7558.692071] Lustre: Skipped 2 previous similar messages [ 7561.674696] Lustre: server umount lustre-MDT0000 complete [ 7563.751282] LustreError: 260541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.771723] LustreError: 260541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7565.600049] LustreError: 261292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245871 with bad export cookie 7211108440950345228 [ 7565.612046] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.619958] LustreError: 261292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7566.228874] Lustre: server umount lustre-MDT0001 complete [ 7576.856231] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 7579.617055] Key type lgssc unregistered [ 7579.914866] LNet: 263893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7579.921699] LNetError: 263893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7580.966558] LNet: Removed LNI 192.168.203.113@tcp [ 7581.954605] Key type .llcrypt unregistered [ 7581.957425] Key type ._llcrypt unregistered [ 7593.104472] Key type ._llcrypt registered [ 7593.106817] Key type .llcrypt registered [ 7593.193592] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7608.757139] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 7623.594809] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 7624.706663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7624.741056] alg: No test for adler32 (adler32-zlib) [ 7625.775746] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7626.035047] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 7627.719671] Key type lgssc registered [ 7629.086445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7666.351544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7666.388835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7667.778592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7667.841780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7667.980488] Lustre: lustre-MDT0000: new disk, initializing [ 7668.126985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.142397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7672.372220] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7683.741286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7683.831273] Lustre: 267397:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7683.884446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7683.887888] Lustre: Skipped 1 previous similar message [ 7683.955288] Lustre: lustre-MDT0001: new disk, initializing [ 7684.085955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7684.122099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7684.141112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7688.692843] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7695.931672] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7699.705848] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7706.639171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7706.938712] Lustre: lustre-OST0000: new disk, initializing [ 7706.942201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7706.948385] Lustre: 269139:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7707.048866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.326941] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7715.908765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7715.920547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7715.998266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7721.559653] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7725.662340] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7725.884082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7730.090688] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7730.358295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7736.801226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7736.805501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.810665] LustreError: Skipped 1 previous similar message [ 7736.812650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7736.842110] Lustre: Skipped 1 previous similar message [ 7738.221276] Lustre: server umount lustre-OST0000 complete [ 7741.927526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.937197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7741.940394] Lustre: Skipped 1 previous similar message [ 7741.957756] Lustre: Skipped 2 previous similar messages [ 7747.041749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7747.054178] Lustre: Skipped 1 previous similar message [ 7747.878507] Lustre: server umount lustre-MDT0000 complete [ 7751.332372] LustreError: 268176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246057 with bad export cookie 8844965537410288413 [ 7751.342063] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7751.345343] LustreError: 268176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7751.722882] Lustre: server umount lustre-MDT0001 complete [ 7760.218688] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:34:23 (1781246063) [ 7768.593767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7769.564666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7774.228968] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7774.694320] LustreError: 271404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.712809] LustreError: 271404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7779.810644] LustreError: 271403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.932058] LustreError: 271404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7786.942974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.350552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7793.708332] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7805.192893] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7810.197148] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7818.046797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7818.475972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.722409] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 7833.303448] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7855.594503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7855.601847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7855.602398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7855.641545] Lustre: Skipped 1 previous similar message [ 7856.948781] Lustre: server umount lustre-OST0000 complete [ 7864.292961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7864.306193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.342180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7864.354340] Lustre: Skipped 1 previous similar message [ 7868.222930] Lustre: server umount lustre-MDT0000 complete [ 7870.946878] LustreError: 273556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.968503] LustreError: 273556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7872.960645] LustreError: 271384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246178 with bad export cookie 8844965537410289295 [ 7872.962613] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7872.973273] LustreError: 271384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7873.335604] Lustre: server umount lustre-MDT0001 complete [ 7886.317275] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 7890.169736] Key type lgssc unregistered [ 7890.624928] LNet: 274612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7890.637799] LNetError: 274612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7890.684330] LNet: Removed LNI 192.168.203.113@tcp [ 7891.958064] Key type .llcrypt unregistered [ 7891.961907] Key type ._llcrypt unregistered [ 7922.415720] Key type ._llcrypt registered [ 7922.419330] Key type .llcrypt registered [ 7922.549626] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 7938.339928] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 7939.558653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7939.666181] alg: No test for adler32 (adler32-zlib) [ 7940.800307] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7941.136922] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 7942.823164] Key type lgssc registered [ 7944.169543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7987.319981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7987.348043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.596558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7988.631935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7988.730326] Lustre: lustre-MDT0000: new disk, initializing [ 7988.811139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.825444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7993.305951] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8003.933980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.001866] Lustre: 278407:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8004.103647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8004.106750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8004.126712] Lustre: Skipped 1 previous similar message [ 8004.192332] Lustre: lustre-MDT0001: new disk, initializing [ 8004.269390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8004.303784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8004.309873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8009.002377] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8017.300329] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8022.561931] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8030.619876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8031.006475] Lustre: lustre-OST2710: new disk, initializing [ 8031.011687] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8031.018619] Lustre: 280148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8031.160040] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8036.916291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8036.922185] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8037.041090] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8038.797389] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8046.883528] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8053.639632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.862544] Lustre: lustre-OST03e8: new disk, initializing [ 8053.870298] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8053.875987] Lustre: 281099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8053.992195] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8059.942414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8059.954350] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8060.089030] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8060.222904] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8067.624410] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8078.447375] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8078.919911] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8086.281318] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8086.710622] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8092.276081] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8092.484025] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8097.220979] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8097.648886] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8103.292625] Lustre: server umount lustre-MDT0000 complete [ 8106.468398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.480913] Lustre: Skipped 2 previous similar messages [ 8107.253807] LustreError: 278398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246413 with bad export cookie 6856949059121439734 [ 8107.262810] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.263469] LustreError: 278398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8107.640911] Lustre: server umount lustre-MDT0001 complete [ 8123.004025] Lustre: server umount lustre-OST2710 complete [ 8137.052489] Lustre: server umount lustre-OST03e8 complete [ 8155.258938] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 8165.098934] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 8202.929484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8203.146629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8203.177307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8203.242914] Lustre: lustre-MDT0000: new disk, initializing [ 8203.313496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.339237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8207.374455] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8216.620531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.697700] Lustre: 286139:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8216.743652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8216.753450] Lustre: Skipped 1 previous similar message [ 8216.860894] Lustre: lustre-MDT0001: new disk, initializing [ 8216.955062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8216.999075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8217.020224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8221.839301] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8228.494891] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8231.749990] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8238.756450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8238.999121] Lustre: lustre-OST0000: new disk, initializing [ 8239.004059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8239.008873] Lustre: 287879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8239.086618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8246.753605] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8246.872367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8246.897348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8247.050836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8256.456496] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8261.903550] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8262.355950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8271.932613] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8272.172698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8277.984788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8277.987218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.007155] LustreError: Skipped 1 previous similar message [ 8278.008689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8278.043900] Lustre: Skipped 2 previous similar messages [ 8281.058390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8281.084259] Lustre: Skipped 1 previous similar message [ 8281.559020] Lustre: server umount lustre-OST0000 complete [ 8288.736627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8288.751510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.770400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.312688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8293.426758] Lustre: server umount lustre-MDT0000 complete [ 8296.422671] LustreError: 288993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8296.451101] LustreError: 288993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8297.763980] LustreError: 286130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246603 with bad export cookie 6856949059121441414 [ 8297.779860] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8298.374851] Lustre: server umount lustre-MDT0001 complete [ 8307.831700] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:43:31 (1781246611) [ 8310.474711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8313.792374] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:43:35 (1781246615) [ 8318.367264] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 8351.708807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.131356] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8352.145963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8352.330144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.651625] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8364.418763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.525326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8364.554354] Lustre: 291580:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8369.239479] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8391.475483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8391.790547] Lustre: Denying initial registration attempt from nid 192.168.203.113@tcp, specified as failover [ 8391.802684] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8391.812777] LustreError: 292998:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8391.819395] LustreError: 292998:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8391.823588] LustreError: 292998:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8391.870710] Lustre: server umount lustre-OST0000 complete [ 8391.877195] LustreError: 292998:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8400.873431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8400.881598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.895357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8400.900668] Lustre: Skipped 2 previous similar messages [ 8404.177920] Lustre: server umount lustre-MDT0000 complete [ 8406.509767] LustreError: 291591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8406.524931] LustreError: 291591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8407.916134] LustreError: 291572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246713 with bad export cookie 6856949059121442296 [ 8407.929691] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8407.934278] LustreError: 291572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8408.252857] Lustre: server umount lustre-MDT0001 complete [ 8418.262199] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 8420.931381] Key type lgssc unregistered [ 8421.215698] LNet: 294105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8421.226794] LNetError: 294105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8421.255521] LNet: Removed LNI 192.168.203.113@tcp [ 8422.304143] Key type .llcrypt unregistered [ 8422.310284] Key type ._llcrypt unregistered [ 8433.821307] Key type ._llcrypt registered [ 8433.824206] Key type .llcrypt registered [ 8433.958446] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 8435.462754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8435.505736] alg: No test for adler32 (adler32-zlib) [ 8436.618094] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 8436.855289] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 8438.495186] Key type lgssc registered [ 8439.965677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8489.365906] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 8500.239197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8500.255786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.650202] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8501.685937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8501.846595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8507.510953] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8518.578779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8518.741537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8518.781823] Lustre: 296835:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8519.082584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8524.491593] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8531.427160] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8536.334825] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8556.342610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8556.771523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8557.176260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.170732] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8573.010584] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8575.460692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8575.462276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8575.482802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8575.486058] Lustre: Skipped 1 previous similar message [ 8580.581583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8580.607617] Lustre: Skipped 1 previous similar message [ 8585.704084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8585.721438] Lustre: Skipped 2 previous similar messages [ 8589.279140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8589.361809] Lustre: server umount lustre-OST0000 complete [ 8595.942556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8595.945560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.963037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8595.964909] Lustre: Skipped 1 previous similar message [ 8595.977110] Lustre: Skipped 1 previous similar message [ 8599.746860] Lustre: server umount lustre-MDT0000 complete [ 8603.546041] LustreError: 296828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781246909 with bad export cookie 4142500588302743789 [ 8603.571927] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8603.876188] Lustre: server umount lustre-MDT0001 complete [ 8614.978061] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 8617.489545] Key type lgssc unregistered [ 8617.785410] LNet: 300053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8617.798085] LNetError: 300053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8617.819967] LNet: Removed LNI 192.168.203.113@tcp [ 8618.615948] Key type .llcrypt unregistered [ 8618.620379] Key type ._llcrypt unregistered [ 8644.955743] Key type ._llcrypt registered [ 8644.958263] Key type .llcrypt registered [ 8645.036358] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 8645.791270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8645.920127] alg: No test for adler32 (adler32-zlib) [ 8647.316536] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 8647.533152] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 8649.207418] Key type lgssc registered [ 8650.328260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8657.861265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8657.887172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8659.460990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.173166] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8669.668557] LustreError: 301453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8669.685976] LustreError: 301453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8671.716239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.009840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8675.936794] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8681.769977] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8685.353959] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8694.572745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8695.221540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.724785] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8708.053924] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8718.308438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8718.313767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.338276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.873726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.875832] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.883711] Lustre: Skipped 1 previous similar message [ 8719.904163] Lustre: Skipped 1 previous similar message [ 8721.084892] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.13@tcp (stopping) [ 8721.090042] Lustre: Skipped 1 previous similar message [ 8722.921287] Lustre: server umount lustre-MDT0000 complete [ 8724.961483] LustreError: 301448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.990198] LustreError: 301448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8726.172499] LustreError: 301450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8727.229134] LustreError: 301435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247033 with bad export cookie 17731132914020741938 [ 8727.238708] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8730.089936] LustreError: 301448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.094048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.149764] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8733.848906] Lustre: server umount lustre-MDT0001 complete [ 8740.704413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8754.254750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.195466] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf611a37115719107 [ 8772.215844] Lustre: MGC192.168.203.113@tcp: Connection restored to 0@lo (at 0@lo) [ 8772.738494] LustreError: 303059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8772.799348] LustreError: 303059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8772.921584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.048150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8779.100930] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8782.323211] LustreError: 303060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.355829] LustreError: 303060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8787.448622] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8789.049657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.383553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8794.453678] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8794.612186] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8801.329520] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8802.965476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8808.358236] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8810.517529] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8819.168614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8819.182414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.198117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8819.202452] Lustre: Skipped 2 previous similar messages [ 8820.199249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8821.577632] Lustre: server umount lustre-OST0000 complete [ 8830.433950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.435373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8830.451062] Lustre: Skipped 1 previous similar message [ 8830.455874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8830.455883] Lustre: Skipped 2 previous similar messages [ 8831.943373] Lustre: server umount lustre-MDT0000 complete [ 8835.416272] LustreError: 301433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247141 with bad export cookie 17731132914020749575 [ 8835.420548] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8835.427749] LustreError: 301433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8835.858571] Lustre: server umount lustre-MDT0001 complete [ 8845.284720] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 8848.112517] Key type lgssc unregistered [ 8848.478172] LNet: 307143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8848.494149] LNetError: 307143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8848.507198] LNet: Removed LNI 192.168.203.113@tcp [ 8849.319838] Key type .llcrypt unregistered [ 8849.323875] Key type ._llcrypt unregistered [ 8875.764499] Key type ._llcrypt registered [ 8875.767959] Key type .llcrypt registered [ 8875.886447] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 8876.681588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8876.772272] alg: No test for adler32 (adler32-zlib) [ 8878.052293] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 8878.390382] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 8880.087199] Key type lgssc registered [ 8881.625425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8892.550234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8892.584086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.442237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.767991] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8904.673598] LustreError: 308543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.706051] LustreError: 308543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8908.499775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.041831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8914.142659] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8919.534576] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8924.218322] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8929.763640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8929.774650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8929.785347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8929.806285] Lustre: Skipped 1 previous similar message [ 8933.924942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8933.933970] Lustre: Skipped 1 previous similar message [ 8934.235459] Lustre: server umount lustre-MDT0000 complete [ 8938.735349] LustreError: 308521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247244 with bad export cookie 4094825204533321784 [ 8938.737599] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8938.742527] LustreError: 308521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8939.094216] Lustre: server umount lustre-MDT0001 complete [ 8953.719415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.947850] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8953.987507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8954.168503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8958.269816] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8966.779406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.967163] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8967.017871] Lustre: 310904:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8967.236733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8972.387507] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 8979.769381] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8985.298322] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8995.766705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.939748] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8995.960588] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8995.980590] LustreError: 312565:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 8995.992496] LustreError: 312565:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8996.010401] LustreError: 312565:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8996.141832] Lustre: server umount lustre-OST0000 complete [ 8996.145184] LustreError: 312565:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9008.680498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.000435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9009.353084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9015.926049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9016.512563] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9021.044315] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9027.788847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9027.873919] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9027.951959] Lustre: lustre-OST0001: new disk, initializing [ 9027.955812] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9027.961860] Lustre: 313992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9028.040267] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9033.255530] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9036.319056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9036.342453] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9036.422579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9038.141944] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9041.382529] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9041.388476] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9041.391891] Lustre: Skipped 2 previous similar messages [ 9041.400051] Lustre: Skipped 1 previous similar message [ 9042.864887] Lustre: server umount lustre-OST0001 complete [ 9046.499466] LustreError: 313033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9046.501423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9046.509508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9046.511580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9046.519038] LustreError: 313033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9046.563921] Lustre: Skipped 1 previous similar message [ 9051.620368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.622884] LustreError: 313032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9051.628954] Lustre: Skipped 2 previous similar messages [ 9051.648027] LustreError: 313032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9052.420960] Lustre: server umount lustre-OST0000 complete [ 9056.737422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.749930] Lustre: Skipped 1 previous similar message [ 9061.860603] LustreError: 311773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.874722] LustreError: 311773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9061.928426] Lustre: server umount lustre-MDT0000 complete [ 9064.915604] LustreError: 310897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247370 with bad export cookie 4094825204533322344 [ 9064.922859] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.925763] LustreError: 310897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9065.149689] Lustre: server umount lustre-MDT0001 complete [ 9073.365757] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 9075.445373] Key type lgssc unregistered [ 9075.663710] LNet: 315818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9075.668850] LNetError: 315818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9075.681284] LNet: Removed LNI 192.168.203.113@tcp [ 9076.257408] Key type .llcrypt unregistered [ 9076.260426] Key type ._llcrypt unregistered [ 9135.557389] Key type ._llcrypt registered [ 9135.560165] Key type .llcrypt registered [ 9135.671299] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 9147.974740] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 9148.708151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9148.719367] alg: No test for adler32 (adler32-zlib) [ 9149.716820] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 9149.937899] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 9151.647181] Key type lgssc registered [ 9152.607939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9178.920991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9178.944802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.155834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9180.180234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9180.234277] Lustre: lustre-MDT0000: new disk, initializing [ 9180.310152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9180.322435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9183.483161] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9191.525115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.634464] Lustre: 320706:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9191.706694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9191.712831] Lustre: Skipped 1 previous similar message [ 9191.812513] Lustre: lustre-MDT0001: new disk, initializing [ 9191.912354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9191.962985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9191.978238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9195.229300] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9200.746448] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9203.297728] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9208.260726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9208.451906] Lustre: lustre-OST0000: new disk, initializing [ 9208.457735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9208.469735] Lustre: 322449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9208.549995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.256716] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9215.018637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9215.024985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9215.074895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9217.727446] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9220.356809] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9220.521503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9223.077395] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9223.200960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9225.189138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.190806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9225.196785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9225.204476] Lustre: Skipped 1 previous similar message [ 9230.308104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9230.314564] Lustre: Skipped 1 previous similar message [ 9230.612211] Lustre: server umount lustre-OST0000 complete [ 9235.424927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9235.427821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9235.436629] Lustre: Skipped 2 previous similar messages [ 9239.176133] Lustre: server umount lustre-MDT0000 complete [ 9241.203325] LustreError: 320697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247547 with bad export cookie 7668488818542799887 [ 9241.204036] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9241.209608] LustreError: 320697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9241.399361] Lustre: server umount lustre-MDT0001 complete [ 9246.757428] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:59:10 (1781247550) [ 9251.480904] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:59:15 (1781247555) [ 9256.937025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9257.234406] LustreError: 325052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.242855] LustreError: 325052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9257.277492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.812848] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9262.561818] LustreError: 325053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.212808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.480190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9268.007907] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9271.770356] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9274.173671] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9278.348472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9278.597525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9282.237808] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9285.497789] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9295.597597] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9296.848881] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9297.714499] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9298.651912] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9299.688697] Lustre: Failing over lustre-MDT0000 [ 9299.858687] Lustre: server umount lustre-MDT0000 complete [ 9301.477314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9301.478983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9301.480669] LustreError: 325047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9301.483855] Lustre: Skipped 1 previous similar message [ 9301.490666] LustreError: Skipped 1 previous similar message [ 9306.593718] LustreError: 325053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9306.600601] LustreError: 325053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9311.712596] LustreError: 325071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9311.722715] LustreError: 325071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9315.167265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.247926] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9315.429735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9315.481618] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9317.929567] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9318.559643] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9320.936291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9320.963873] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9321.001142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9323.050198] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9324.035683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9326.667178] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9327.711529] Lustre: Failing over lustre-MDT0000 [ 9327.895534] Lustre: server umount lustre-MDT0000 complete [ 9328.811780] LustreError: 325048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.827204] LustreError: 325048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9331.170196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9331.172889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9331.188069] Lustre: Skipped 2 previous similar messages [ 9342.418480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.500284] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.700114] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9344.158558] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9344.782955] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9348.071572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9348.074536] Lustre: Skipped 2 previous similar messages [ 9348.092812] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9348.114682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9350.080933] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9350.995571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9358.306539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.306925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9358.307795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9358.307802] Lustre: Skipped 1 previous similar message [ 9358.316503] Lustre: Skipped 2 previous similar messages [ 9360.130155] Lustre: server umount lustre-MDT0000 complete [ 9361.885956] LustreError: 325032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247667 with bad export cookie 7668488818542802554 [ 9361.888714] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.893031] LustreError: 325032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9362.065097] Lustre: server umount lustre-MDT0001 complete [ 9374.140210] Lustre: server umount lustre-OST0000 complete [ 9382.388073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.660306] LustreError: 330826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.670845] LustreError: 330826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 9382.700800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9382.704152] Lustre: Skipped 1 previous similar message [ 9384.565459] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9388.559952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.582512] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9392.975432] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9394.657809] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9397.990900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9401.020945] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9403.766191] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9407.400038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9412.579825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.579915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9412.583524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.583532] Lustre: Skipped 2 previous similar messages [ 9412.589104] Lustre: Skipped 1 previous similar message [ 9417.057247] Lustre: server umount lustre-OST0000 complete [ 9422.816767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9422.819303] Lustre: Skipped 2 previous similar messages [ 9433.055112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9433.131578] Lustre: server umount lustre-MDT0000 complete [ 9435.019785] LustreError: 330808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247740 with bad export cookie 7668488818542803303 [ 9435.025916] LustreError: 330808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9435.026575] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9435.232758] Lustre: server umount lustre-MDT0001 complete [ 9440.741976] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 9442.180790] Key type lgssc unregistered [ 9442.349312] LNet: 333999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9442.359662] LNetError: 333999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9442.369337] LNet: Removed LNI 192.168.203.113@tcp [ 9442.770259] Key type .llcrypt unregistered [ 9442.772240] Key type ._llcrypt unregistered [ 9457.383146] Key type ._llcrypt registered [ 9457.385096] Key type .llcrypt registered [ 9457.437764] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 9457.911948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9457.958550] alg: No test for adler32 (adler32-zlib) [ 9458.886860] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 9459.021918] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 9460.639193] Key type lgssc registered [ 9461.228377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9465.147712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9465.156046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.405949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9468.257936] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9472.282425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.464576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9474.432150] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9477.065213] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9478.978373] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9482.240393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9482.415256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9484.925963] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9487.433257] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9491.627695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9502.180750] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.183249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9502.189347] Lustre: Skipped 1 previous similar message [ 9502.198422] Lustre: Skipped 2 previous similar messages [ 9504.367811] Lustre: server umount lustre-MDT0000 complete [ 9506.186141] LustreError: 336197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247812 with bad export cookie 5026589541360643232 [ 9506.188762] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9506.192852] LustreError: 336197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9506.362904] Lustre: server umount lustre-MDT0001 complete [ 9518.585352] Lustre: server umount lustre-OST0000 complete [ 9525.589671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.873850] LustreError: 338848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9525.888635] LustreError: 338848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9525.913150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9528.100588] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9531.360318] LustreError: 338849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9532.619702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.788945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9534.704333] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9537.495516] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9539.589912] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9542.913851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9543.011029] LustreError: 340440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.086235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9545.815133] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9548.457795] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9550.317279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9563.616653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9563.620181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.626029] Lustre: Skipped 1 previous similar message [ 9563.628648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9565.667170] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.667174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9565.678223] Lustre: Skipped 1 previous similar message [ 9567.978632] Lustre: server umount lustre-MDT0000 complete [ 9569.773154] LustreError: 338828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247875 with bad export cookie 5026589541360644352 [ 9569.774071] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9569.780722] LustreError: 338828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9569.924298] Lustre: server umount lustre-MDT0001 complete [ 9582.065048] Lustre: server umount lustre-OST0000 complete [ 9588.602676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.835332] LustreError: 342418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.871597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9590.725760] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9594.336296] LustreError: 342419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.636846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.094744] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9607.178541] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9613.018595] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9620.591590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9620.960688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9620.968292] Lustre: Skipped 1 previous similar message [ 9628.841277] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9633.259156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9635.895317] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9643.511904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9643.527109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9643.558920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9643.572413] Lustre: Skipped 2 previous similar messages [ 9645.541502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9648.762608] Lustre: server umount lustre-OST0000 complete [ 9655.777396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9655.805806] Lustre: Skipped 2 previous similar messages [ 9655.807842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9655.838061] Lustre: Skipped 3 previous similar messages [ 9661.994333] Lustre: server umount lustre-MDT0000 complete [ 9666.028984] LustreError: 343129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.052156] LustreError: 343129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9669.383904] LustreError: 342398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781247975 with bad export cookie 5026589541360645360 [ 9669.421647] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9669.431277] LustreError: 342398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9669.962057] Lustre: server umount lustre-MDT0001 complete [ 9683.684016] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 9686.253772] Key type lgssc unregistered [ 9686.521782] LNet: 345594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9686.539317] LNetError: 345594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9686.554946] LNet: Removed LNI 192.168.203.113@tcp [ 9687.346460] Key type .llcrypt unregistered [ 9687.348395] Key type ._llcrypt unregistered [ 9709.674614] Key type ._llcrypt registered [ 9709.676385] Key type .llcrypt registered [ 9709.761220] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 9710.712413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9710.751779] alg: No test for adler32 (adler32-zlib) [ 9711.778643] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 9712.063741] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 9713.767164] Key type lgssc registered [ 9714.872714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9721.416073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9721.425919] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9721.431888] LustreError: 347018:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9721.441373] LustreError: 347018:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9721.445869] LustreError: 347018:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9721.450912] LustreError: 347018:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9721.457158] LustreError: 347018:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9727.743751] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.750766] LustreError: 347415:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9727.766914] LustreError: 347415:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9727.774496] LustreError: 347415:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9727.780335] LustreError: 347415:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9727.786736] LustreError: 347415:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9743.701248] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [ 9746.304714] Key type lgssc unregistered [ 9746.594431] LNet: 348401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9746.613353] LNetError: 348401:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9746.625910] LNet: Removed LNI 192.168.203.113@tcp [ 9747.274888] Key type .llcrypt unregistered [ 9747.276688] Key type ._llcrypt unregistered [ 9762.879264] Key type ._llcrypt registered [ 9762.881421] Key type .llcrypt registered [ 9762.962382] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [ 9776.407244] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [ 9777.434981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9777.468742] alg: No test for adler32 (adler32-zlib) [ 9778.443908] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 9778.602091] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [ 9780.297263] Key type lgssc registered [ 9781.405840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9810.896557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9810.976055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9812.260032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9812.302145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9812.402317] Lustre: lustre-MDT0000: new disk, initializing [ 9812.497608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9812.522031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9816.599991] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9825.753051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.853873] Lustre: 351521:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9825.919597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9825.926438] Lustre: Skipped 1 previous similar message [ 9826.003680] Lustre: lustre-MDT0001: new disk, initializing [ 9826.116473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9826.144178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9826.159098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9829.859474] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9836.413853] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9839.396226] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9845.310578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9845.537365] Lustre: lustre-OST0000: new disk, initializing [ 9845.540817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9845.546411] Lustre: 353259:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9845.619733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9851.005772] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9852.496481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9852.513427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9852.632445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9857.974815] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9862.633952] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9862.907972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9867.211931] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9867.392848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9873.376681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9873.377103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.394989] LustreError: Skipped 1 previous similar message [ 9873.398714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9873.407754] Lustre: Skipped 1 previous similar message [ 9873.424628] Lustre: Skipped 1 previous similar message [ 9875.397595] Lustre: server umount lustre-OST0000 complete [ 9882.592470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9882.597097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.618722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9883.618152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9883.648793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9883.660506] Lustre: Skipped 1 previous similar message [ 9888.742730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9888.745591] Lustre: Skipped 1 previous similar message [ 9893.343996] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9893.508169] Lustre: server umount lustre-MDT0000 complete [ 9896.296060] LustreError: 351512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248202 with bad export cookie 13928845381508354164 [ 9896.299111] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9896.306747] LustreError: 351512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9896.530894] Lustre: server umount lustre-MDT0001 complete [ 9903.733712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9904.983561] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:10:08 (1781248208) [ 9911.424276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.930908] LustreError: 355574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9911.954927] LustreError: 355574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9912.025288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.907993] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9917.410376] LustreError: 355575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.437341] LustreError: 355575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9921.509105] LustreError: 355574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.370503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.655176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9927.947904] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9931.949508] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9934.736668] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9939.531599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9939.752657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9944.219822] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9948.394873] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9954.893841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9955.017066] Lustre: lustre-OST0001: new disk, initializing [ 9955.020213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9955.024385] Lustre: 358068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9958.526838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9958.539269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9958.615810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9962.608752] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [ 9970.402816] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9979.081573] Lustre: lustre-OST0001: Not available for connect from 192.168.203.13@tcp (stopping) [ 9979.298986] Lustre: server umount lustre-OST0001 complete [ 9979.366083] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9979.367681] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.374244] LustreError: 357169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9979.375015] LustreError: Skipped 1 previous similar message [ 9979.440801] Lustre: Skipped 1 previous similar message [ 9984.159156] LustreError: 357168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9984.222378] LustreError: 357168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9994.721597] LustreError: 357166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9994.759697] LustreError: 357166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9999.844843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.859774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10006.017280] Lustre: server umount lustre-OST0000 complete [10010.088217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.111040] Lustre: Skipped 2 previous similar messages [10015.551331] Lustre: server umount lustre-MDT0000 complete [10019.255308] LustreError: 355556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248325 with bad export cookie 13928845381508355046 [10019.256648] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10019.267089] LustreError: 355556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10025.624107] Lustre: server umount lustre-MDT0001 complete [10035.246757] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [10038.238655] Key type lgssc unregistered [10038.495191] LNet: 359925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10038.513188] LNetError: 359925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10038.546121] LNet: Removed LNI 192.168.203.113@tcp [10039.362244] Key type .llcrypt unregistered [10039.373789] Key type ._llcrypt unregistered [10086.843459] Key type ._llcrypt registered [10086.845879] Key type .llcrypt registered [10086.951808] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [10088.437293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10088.476735] alg: No test for adler32 (adler32-zlib) [10089.582285] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [10089.878683] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [10091.519261] Key type lgssc registered [10092.542350] Lustre: Echo OBD driver; http://www.lustre.org/ [10099.941608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10099.976815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10101.355824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10101.378921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10101.516765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10105.591229] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10114.561774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10114.674922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10114.728838] Lustre: 362685:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10115.041152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10118.983145] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10125.481553] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10129.288993] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10134.497675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.517050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10134.527181] Lustre: Skipped 1 previous similar message [10139.617743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10144.739753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10144.748381] Lustre: Skipped 2 previous similar messages [10146.272615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10146.497894] LustreError: 364052:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (0000000097ef8650) refcount nonzero (1) after lock cleanup; forcing cleanup. [10146.587717] Lustre: server umount lustre-MDT0000 complete [10151.136963] LustreError: 362679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248457 with bad export cookie 2473159200055344360 [10151.151678] LustreError: 362679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10151.156944] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10151.518974] Lustre: server umount lustre-MDT0001 complete [10162.260876] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:14:25 (1781248465) [10168.797822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10169.287140] LustreError: 364972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.310349] LustreError: 364972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10169.338541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10173.690602] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10174.432859] LustreError: 364973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10179.554576] LustreError: 364972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10181.440436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.766820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10185.711610] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10190.624870] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10194.823585] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10201.398956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10201.574209] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10201.769714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10208.411133] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10213.762353] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10220.948428] Lustre: Permanently deactivating lustre-OST0000 [10220.972088] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10229.280536] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10230.842657] LustreError: 367621:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10230.854425] LustreError: 367621:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10230.861369] LustreError: 367621:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10234.007733] LustreError: 367670:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10234.021734] LustreError: 367670:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10242.317886] Lustre: server umount lustre-OST0000 complete [10245.960799] LustreError: 367919:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10245.967235] LustreError: 367919:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10250.218650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.244669] Lustre: Skipped 1 previous similar message [10250.256519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10250.282836] Lustre: Skipped 1 previous similar message [10255.330358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.640938] Lustre: server umount lustre-MDT0000 complete [10260.451399] LustreError: 364972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.470954] LustreError: 364972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10264.813026] LustreError: 366581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248570 with bad export cookie 2473159200055344941 [10264.834026] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10264.843957] LustreError: 366581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10274.587302] Lustre: server umount lustre-MDT0001 complete [10287.030433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.972579] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10295.078752] LustreError: 369062:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10301.769861] Lustre: 369401:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.113@tcp [10304.103852] Lustre: server umount MGS complete [10317.882690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.828046] LustreError: 370125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.902230] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10318.919938] Lustre: Skipped 1 previous similar message [10319.055434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10325.868932] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10329.568397] LustreError: 370125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10329.605494] LustreError: 370125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10337.963375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10338.379159] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10338.523888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10344.044925] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10350.038241] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10354.130032] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10356.131924] Lustre: Permanently reactivating lustre-OST0000 [10356.147925] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10356.158265] Lustre: Skipped 2 previous similar messages [10384.352115] LustreError: 371687:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10393.591230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10394.061718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10400.480983] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10406.418627] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10413.541601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10413.549393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.563097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10413.571266] Lustre: Skipped 1 previous similar message [10414.210537] Lustre: server umount lustre-OST0000 complete [10419.680921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10429.015700] Lustre: server umount lustre-MDT0000 complete [10429.923643] LustreError: 372042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10429.939891] LustreError: 372042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10432.834757] LustreError: 370843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248738 with bad export cookie 2473159200055345991 [10432.837323] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10432.849352] LustreError: 370843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10433.305370] Lustre: server umount lustre-MDT0001 complete [10447.420969] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [10450.795617] Key type lgssc unregistered [10451.127339] LNet: 374184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10451.134274] LNetError: 374184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10451.161932] LNet: Removed LNI 192.168.203.113@tcp [10452.097185] Key type .llcrypt unregistered [10452.100275] Key type ._llcrypt unregistered [10471.439611] Key type ._llcrypt registered [10471.441370] Key type .llcrypt registered [10471.565185] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [10486.253494] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [10487.323789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10487.344701] alg: No test for adler32 (adler32-zlib) [10488.359926] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [10488.534850] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [10490.191179] Key type lgssc registered [10491.260497] Lustre: Echo OBD driver; http://www.lustre.org/ [10526.264735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10526.282608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.599166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10527.637245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10527.737947] Lustre: lustre-MDT0000: new disk, initializing [10527.844404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10527.868957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10531.624247] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10541.536450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.667991] Lustre: 377308:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10541.725363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10541.732669] Lustre: Skipped 1 previous similar message [10541.842546] Lustre: lustre-MDT0001: new disk, initializing [10541.950350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10541.982586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10541.996124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10546.093106] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10553.903277] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10557.148420] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10562.850661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10563.081800] Lustre: lustre-OST0000: new disk, initializing [10563.085129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10563.093914] Lustre: 379049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10563.190607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10568.517471] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10570.289089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10570.308941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10570.395029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10575.650937] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10579.793715] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10580.009520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10583.762412] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10583.931798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10586.081433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10586.088047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.093046] LustreError: Skipped 1 previous similar message [10586.116270] Lustre: Skipped 1 previous similar message [10586.117884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10586.127908] Lustre: Skipped 1 previous similar message [10591.211937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10591.222331] Lustre: Skipped 1 previous similar message [10591.905782] Lustre: server umount lustre-OST0000 complete [10596.321393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10596.323178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10596.336845] Lustre: Skipped 1 previous similar message [10603.237991] Lustre: server umount lustre-MDT0000 complete [10606.309435] LustreError: 377302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781248912 with bad export cookie 4943648444648556441 [10606.310204] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10606.323588] LustreError: 377302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10606.624755] Lustre: server umount lustre-MDT0001 complete [10613.467758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10614.917857] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:21:58 (1781248918) [10620.367523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.651823] LustreError: 381361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.676691] LustreError: 381361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10620.712233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10624.559932] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10626.017837] LustreError: 381362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.120841] LustreError: 381361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.812960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10632.165595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10635.659579] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10640.078784] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10643.489047] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10649.263972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10649.596310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10654.275207] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10658.720649] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10660.259741] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10666.920327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10673.124921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10673.132720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10673.143089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10673.146239] Lustre: Skipped 1 previous similar message [10676.197189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.032023] Lustre: server umount lustre-OST0000 complete [10681.320672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.328687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10681.334516] Lustre: Skipped 3 previous similar messages [10691.557565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10691.564869] Lustre: Skipped 3 previous similar messages [10695.135726] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10695.271303] Lustre: server umount lustre-MDT0000 complete [10696.677281] LustreError: 381362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.706664] LustreError: 381362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10698.388571] LustreError: 381342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249004 with bad export cookie 4943648444648557323 [10698.391664] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10698.398214] LustreError: 381342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10698.858633] Lustre: server umount lustre-MDT0001 complete [10706.765194] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [10709.069711] Key type lgssc unregistered [10709.350058] LNet: 384728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10709.359414] LNetError: 384728:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10709.374744] LNet: Removed LNI 192.168.203.113@tcp [10710.145562] Key type .llcrypt unregistered [10710.148533] Key type ._llcrypt unregistered [10728.725832] Key type ._llcrypt registered [10728.728692] Key type .llcrypt registered [10728.812892] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [10729.082480] Key type .llcrypt unregistered [10729.085223] Key type ._llcrypt unregistered [10742.919919] Key type ._llcrypt registered [10742.925398] Key type .llcrypt registered [10743.019600] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [10743.732762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10743.799039] alg: No test for adler32 (adler32-zlib) [10744.775614] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [10744.949455] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [10746.592361] Key type lgssc registered [10747.507213] Lustre: Echo OBD driver; http://www.lustre.org/ [10753.125850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10753.138231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10754.496676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10757.506134] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10764.610750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.769882] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.794427] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10764.874140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10769.264436] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10769.892147] LustreError: 386885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.906386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10769.909957] LustreError: 386885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10773.332632] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10775.013174] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.027255] LustreError: 386884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10778.677582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10778.969377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10782.180398] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10794.467076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10794.467900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10794.475333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10794.486677] Lustre: Skipped 1 previous similar message [10798.694606] Lustre: server umount lustre-OST0000 complete [10804.708811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.716323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10804.724263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10804.741482] Lustre: Skipped 2 previous similar messages [10807.964709] Lustre: server umount lustre-MDT0000 complete [10809.826615] LustreError: 386885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.839441] LustreError: 386885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10810.923663] LustreError: 386867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249116 with bad export cookie 8327665130870032977 [10810.931453] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10810.934388] LustreError: 386867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10811.369981] Lustre: server umount lustre-MDT0001 complete [10818.877531] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [10821.113580] Key type lgssc unregistered [10821.351659] LNet: 389874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10821.358248] LNetError: 389874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10821.372744] LNet: Removed LNI 192.168.203.113@tcp [10822.094832] Key type .llcrypt unregistered [10822.097525] Key type ._llcrypt unregistered [10840.337601] Key type ._llcrypt registered [10840.340709] Key type .llcrypt registered [10840.430575] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [10841.255812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10841.274416] alg: No test for adler32 (adler32-zlib) [10842.303932] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [10842.510909] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [10844.167164] Key type lgssc registered [10844.973777] Lustre: Echo OBD driver; http://www.lustre.org/ [10851.319178] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10851.339159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10866.911557] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10872.031201] LustreError: 391228:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.113@tcp: failed processing log, type 4: rc = -110 [10898.655202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10902.879898] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10906.937340] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10911.850943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.275135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10912.338089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10915.625172] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10918.371962] LustreError: 392152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.383458] LustreError: 392152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10922.437604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.779092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10927.262974] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [10934.479351] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10938.136927] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10948.085044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10948.092227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.102185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10953.699043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10953.715971] Lustre: Skipped 1 previous similar message [10958.828583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10958.851516] Lustre: Skipped 1 previous similar message [10961.887501] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10962.171171] Lustre: server umount lustre-OST0000 complete [10969.059969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10969.063519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.067266] LustreError: Skipped 1 previous similar message [10969.067888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10969.067892] Lustre: Skipped 1 previous similar message [10969.131515] Lustre: Skipped 2 previous similar messages [10974.177325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10974.186200] Lustre: Skipped 2 previous similar messages [10981.343137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10981.838974] Lustre: server umount lustre-MDT0000 complete [10984.418384] LustreError: 392156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10984.453570] LustreError: 392156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10986.543321] LustreError: 391235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249292 with bad export cookie 5754184547146460843 [10986.548485] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10986.564298] LustreError: 391235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10987.241728] Lustre: server umount lustre-MDT0001 complete [11000.332994] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [11005.358031] Key type lgssc unregistered [11005.949628] LNet: 394441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11005.960533] LNetError: 394441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11005.991819] LNet: Removed LNI 192.168.203.113@tcp [11007.197149] Key type .llcrypt unregistered [11007.198719] Key type ._llcrypt unregistered [11032.428776] Key type ._llcrypt registered [11032.431728] Key type .llcrypt registered [11032.530519] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [11033.556969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11033.703638] alg: No test for adler32 (adler32-zlib) [11034.859468] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11035.160763] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [11036.959746] Key type lgssc registered [11038.237053] Lustre: Echo OBD driver; http://www.lustre.org/ [11045.897364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11045.938042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.384401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11051.248643] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11057.633632] LustreError: 395841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11057.657901] LustreError: 395841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11059.420455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.893596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11063.488704] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11068.401783] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11071.947574] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11077.494658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11077.830134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11083.173557] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11088.548179] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11092.209620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11097.569610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.574800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11097.590486] Lustre: Skipped 2 previous similar messages [11097.606433] Lustre: Skipped 2 previous similar messages [11099.332580] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.13@tcp (stopping) [11102.427857] Lustre: server umount lustre-MDT0000 complete [11102.690145] LustreError: 395842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.742768] LustreError: 395842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11104.714514] LustreError: 395822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.13@tcp arrived at 1781249410 with bad export cookie 14327236262260987704 [11106.716928] LustreError: 395821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249412 with bad export cookie 14327236262260987039 [11106.729717] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11106.734194] LustreError: 395821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11107.809701] LustreError: 395842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11107.816932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11107.828364] LustreError: 395842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11107.863891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.844144] LustreError: 395841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.849508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.857814] LustreError: 395841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11112.948319] Lustre: server umount lustre-OST0000 complete [11116.962114] LustreError: 395842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11116.974642] LustreError: 395842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11118.860364] Lustre: server umount lustre-MDT0001 complete [11128.173375] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [11131.070973] Key type lgssc unregistered [11131.441508] LNet: 399169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11131.466290] LNetError: 399169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11132.534155] LNet: Removed LNI 192.168.203.113@tcp [11133.420112] Key type .llcrypt unregistered [11133.422374] Key type ._llcrypt unregistered [11155.234955] Key type ._llcrypt registered [11155.239324] Key type .llcrypt registered [11155.383186] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [11156.438254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11156.461559] alg: No test for adler32 (adler32-zlib) [11157.621306] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11157.896650] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [11159.647168] Key type lgssc registered [11160.763719] Lustre: Echo OBD driver; http://www.lustre.org/ [11169.232338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11169.258612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.706514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11175.469634] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11181.031122] LustreError: 400569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11181.042436] LustreError: 400569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11183.926677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.248892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11188.351489] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11193.929908] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11196.910506] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11202.587306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11202.814364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11208.765126] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11215.102519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11215.483687] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11220.450312] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11220.485664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11220.487743] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11220.489313] Lustre: Skipped 1 previous similar message [11223.248768] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) [11225.589419] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11225.596336] Lustre: Skipped 1 previous similar message [11226.502903] Lustre: server umount lustre-MDT0001 complete [11228.321808] LustreError: 400565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.693243] LustreError: 400568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.710782] LustreError: 400568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11235.808752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11235.813092] LustreError: 400569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11235.821040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11235.821048] Lustre: Skipped 1 previous similar message [11235.821585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11238.483097] Lustre: server umount lustre-OST0000 complete [11240.929328] LustreError: 400568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11243.781747] Lustre: server umount lustre-MDT0000 complete [11253.942198] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [11255.953932] Key type lgssc unregistered [11256.184728] LNet: 403895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11256.191305] LNetError: 403895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11256.212419] LNet: Removed LNI 192.168.203.113@tcp [11256.819117] Key type .llcrypt unregistered [11256.821270] Key type ._llcrypt unregistered [11278.446643] Key type ._llcrypt registered [11278.449691] Key type .llcrypt registered [11278.555742] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [11292.921286] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [11294.089742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11294.112655] alg: No test for adler32 (adler32-zlib) [11295.128705] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11295.367541] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [11297.047165] Key type lgssc registered [11298.237451] Lustre: Echo OBD driver; http://www.lustre.org/ [11333.181652] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [11347.467907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11347.508645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.903736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11348.956810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11349.048258] Lustre: lustre-MDT0000: new disk, initializing [11349.170687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11349.195583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11353.030343] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11369.031284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11369.304571] Lustre: lustre-OST0000: new disk, initializing [11369.309306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11369.317392] Lustre: Skipped 1 previous similar message [11369.333746] Lustre: 408859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11369.421887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11375.209855] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11378.316982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11378.333275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11378.373527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11382.628840] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11397.370427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11397.498983] Lustre: 407859:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11397.566648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11397.653509] Lustre: lustre-MDT0001: new disk, initializing [11397.786471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11397.846646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11397.857110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11401.914231] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11410.003653] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11410.189506] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11414.414210] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11414.576957] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11419.322888] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) [11422.180515] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11422.183810] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11422.185918] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11422.201579] Lustre: Skipped 1 previous similar message [11424.413603] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.13@tcp (stopping) [11424.425250] Lustre: Skipped 1 previous similar message [11425.401943] Lustre: server umount lustre-MDT0001 complete [11432.417476] LustreError: 407869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.434049] LustreError: 407869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11437.545862] LustreError: 407864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.558059] LustreError: 407864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11442.666811] LustreError: 407869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11442.668507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11442.705531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.778458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.779452] LustreError: 407870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.563333] Lustre: server umount lustre-OST0000 complete [11466.207132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11466.425758] Lustre: server umount lustre-MDT0000 complete [11479.317352] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [11482.217271] Key type lgssc unregistered [11482.523497] LNet: 412427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11482.532608] LNetError: 412427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11482.556844] LNet: Removed LNI 192.168.203.113@tcp [11483.501315] Key type .llcrypt unregistered [11483.502955] Key type ._llcrypt unregistered [11525.885172] Key type ._llcrypt registered [11525.888764] Key type .llcrypt registered [11526.030507] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [11526.292627] Key type .llcrypt unregistered [11526.298630] Key type ._llcrypt unregistered [11541.698919] Key type ._llcrypt registered [11541.704040] Key type .llcrypt registered [11541.834879] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [11542.914804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11542.932844] alg: No test for adler32 (adler32-zlib) [11543.970704] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [11544.246188] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [11545.951186] Key type lgssc registered [11547.076244] Lustre: Echo OBD driver; http://www.lustre.org/ [11570.359715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11570.392955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.670097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11571.694181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11571.796992] Lustre: lustre-MDT0000: new disk, initializing [11571.909548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11571.925941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11575.456909] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11584.348328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11584.452924] Lustre: 415774:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11584.522740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11584.529904] Lustre: Skipped 1 previous similar message [11584.596063] Lustre: lustre-MDT0001: new disk, initializing [11584.691922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11584.754918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11584.766392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11587.911604] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11594.363428] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11597.347850] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11602.710652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11603.018551] Lustre: lustre-OST0000: new disk, initializing [11603.026533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11603.036970] Lustre: 417516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11603.163908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11608.060903] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11610.156366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11610.169589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11610.243625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11614.147472] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11620.322952] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11620.329105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.338708] Lustre: Skipped 2 previous similar messages [11620.353615] Lustre: Skipped 2 previous similar messages [11625.440798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11625.452571] Lustre: Skipped 2 previous similar messages [11626.189778] Lustre: server umount lustre-MDT0000 complete [11629.198897] LustreError: 415767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781249935 with bad export cookie 12603453087435385632 [11629.204645] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11629.209777] LustreError: 415767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11629.471889] Lustre: server umount lustre-MDT0001 complete [11642.691859] Lustre: server umount lustre-OST0000 complete [11649.541550] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:39:13 (1781249953) [11655.327397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11655.765884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11658.894341] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11660.772216] LustreError: 419429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.790679] LustreError: 419429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11665.550181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.833793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11669.647640] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11673.848389] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11677.110207] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11695.864557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11695.973064] Lustre: Found index 0 for lustre-OST0000, updating log [11696.153568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11701.297494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11701.301905] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [11705.760686] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11710.371376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.383522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11715.438470] Lustre: server umount lustre-OST0000 complete [11729.994463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.001433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11730.002719] Lustre: Skipped 1 previous similar message [11730.013045] Lustre: Skipped 3 previous similar messages [11735.008405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11735.024827] Lustre: Skipped 1 previous similar message [11736.317553] Lustre: server umount lustre-MDT0000 complete [11737.057772] LustreError: 420139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.078792] LustreError: 420139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11738.083720] LustreError: 419425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.094846] LustreError: 419425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11739.465354] LustreError: 421042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250045 with bad export cookie 12603453087435387018 [11739.480572] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11739.487246] LustreError: 421042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11739.773862] Lustre: server umount lustre-MDT0001 complete [11746.959909] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:40:50 (1781250050) [11756.255155] LNet: 1 local NIs in recovery (showing 1): 192.168.203.113@tcp [12352.843958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12353.149993] LustreError: 425494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.227684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12356.403483] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12358.630053] LustreError: 425495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12363.001595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12363.282167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12365.962317] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12369.361918] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12371.864329] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12389.870339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12390.015898] Lustre: Found index 0 for lustre-OST0000, updating log [12390.191051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12394.912337] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12398.991657] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12401.450503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12401.465758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12406.556468] Lustre: server umount lustre-OST0000 complete [12408.543643] LNet: 1 local NIs in recovery (showing 1): 192.168.203.113@tcp [12409.314504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12409.318800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.326516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12410.337314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12411.361050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12411.369306] Lustre: Skipped 3 previous similar messages [12415.297851] Lustre: server umount lustre-MDT0000 complete [12415.461537] LustreError: 427201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12415.472134] LustreError: 427201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12417.505770] LustreError: 425489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.520925] LustreError: 425489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12417.549078] LustreError: 427117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250723 with bad export cookie 12603453087435388096 [12417.551895] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12417.562021] LustreError: 427117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12417.739918] Lustre: server umount lustre-MDT0001 complete [12440.535928] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [12447.179629] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [12469.589925] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:52:53 (1781250773) [12479.245805] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [12481.126606] Key type lgssc unregistered [12481.311643] LNet: 431708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12481.315588] LNetError: 431708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12519.205896] LNet: Removed LNI 192.168.203.113@tcp [12519.738646] Key type .llcrypt unregistered [12519.741504] Key type ._llcrypt unregistered [12528.882402] Key type ._llcrypt registered [12528.884421] Key type .llcrypt registered [12528.943479] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [12529.463347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12529.480129] alg: No test for adler32 (adler32-zlib) [12530.416093] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [12530.525658] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [12532.127656] Key type lgssc registered [12532.698150] Lustre: Echo OBD driver; http://www.lustre.org/ [12534.889403] LNet: Added LNI 192.168.203.113@tcp100 [8/256/0/180] [12535.407938] LNet: Added LNI 192.168.203.113@tcp101 [8/256/0/180] [12535.827882] LNet: Added LNI 192.168.203.113@tcp102 [8/256/0/180] [12536.238605] LNet: Added LNI 192.168.203.113@tcp103 [8/256/0/180] [12536.692904] LNet: Added LNI 192.168.203.113@tcp104 [8/256/0/180] [12537.143391] LNet: Added LNI 192.168.203.113@tcp105 [8/256/0/180] [12537.641160] LNet: Added LNI 192.168.203.113@tcp106 [8/256/0/180] [12538.132840] LNet: Added LNI 192.168.203.113@tcp107 [8/256/0/180] [12538.614191] LNet: Added LNI 192.168.203.113@tcp108 [8/256/0/180] [12539.147216] LNet: Added LNI 192.168.203.113@tcp109 [8/256/0/180] [12539.625400] LNet: Added LNI 192.168.203.113@tcp110 [8/256/0/180] [12540.046272] LNet: Added LNI 192.168.203.113@tcp111 [8/256/0/180] [12540.518231] LNet: Added LNI 192.168.203.113@tcp112 [8/256/0/180] [12541.040554] LNet: Added LNI 192.168.203.113@tcp113 [8/256/0/180] [12541.566311] LNet: Added LNI 192.168.203.113@tcp114 [8/256/0/180] [12542.170293] LNet: Added LNI 192.168.203.113@tcp115 [8/256/0/180] [12542.665893] LNet: Added LNI 192.168.203.113@tcp116 [8/256/0/180] [12543.128843] LNet: Added LNI 192.168.203.113@tcp117 [8/256/0/180] [12543.722455] LNet: Added LNI 192.168.203.113@tcp118 [8/256/0/180] [12544.291141] LNet: Added LNI 192.168.203.113@tcp119 [8/256/0/180] [12544.882816] LNet: Added LNI 192.168.203.113@tcp120 [8/256/0/180] [12545.431050] LNet: Added LNI 192.168.203.113@tcp121 [8/256/0/180] [12545.977274] LNet: Added LNI 192.168.203.113@tcp122 [8/256/0/180] [12546.474530] LNet: Added LNI 192.168.203.113@tcp123 [8/256/0/180] [12547.121191] LNet: Added LNI 192.168.203.113@tcp124 [8/256/0/180] [12547.627703] LNet: Added LNI 192.168.203.113@tcp125 [8/256/0/180] [12548.062784] LNet: Added LNI 192.168.203.113@tcp126 [8/256/0/180] [12548.539610] LNet: Added LNI 192.168.203.113@tcp127 [8/256/0/180] [12548.997821] LNet: Added LNI 192.168.203.113@tcp128 [8/256/0/180] [12549.434665] LNet: Added LNI 192.168.203.113@tcp129 [8/256/0/180] [12549.918506] LNet: Added LNI 192.168.203.113@tcp130 [8/256/0/180] [12550.443262] LNet: Added LNI 192.168.203.113@tcp131 [8/256/0/180] [12551.006408] LNet: Added LNI 192.168.203.113@tcp132 [8/256/0/180] [12551.496178] LNet: Added LNI 192.168.203.113@tcp133 [8/256/0/180] [12551.965824] LNet: Added LNI 192.168.203.113@tcp134 [8/256/0/180] [12552.396977] LNet: Added LNI 192.168.203.113@tcp135 [8/256/0/180] [12556.031369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12556.046283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12557.129758] Lustre: 434686:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.113@tcp: can't add NID 192.168.203.113@tcp130: rc = -75 [12557.234103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12557.260493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12557.308417] Lustre: lustre-MDT0000: new disk, initializing [12557.362430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12557.373213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12559.693152] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12566.048734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.133200] Lustre: 434703:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12566.175370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12566.179889] Lustre: Skipped 1 previous similar message [12566.237687] Lustre: lustre-MDT0001: new disk, initializing [12566.257302] Lustre: 435482:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.113@tcp: can't add NID 192.168.203.113@tcp130: rc = -75 [12566.262351] Lustre: 435482:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12566.327990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12566.348643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12566.353357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12568.490116] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12573.345843] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12575.677650] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12579.477934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12579.690916] Lustre: lustre-OST0000: new disk, initializing [12579.695383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12579.702178] Lustre: 436444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12579.757048] Lustre: 436449:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.113@tcp: can't add NID 192.168.203.113@tcp130: rc = -75 [12579.761796] Lustre: 436449:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [12579.797377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12582.960619] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12587.309499] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12587.532824] Lustre: 437042:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.113@tcp: can't add NID 192.168.203.113@tcp130: rc = -75 [12587.537032] Lustre: 437042:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12587.571450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12587.578951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12587.658155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12592.608246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12592.613349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12592.624097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12593.120047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12593.128847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12593.138120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12593.141235] Lustre: Skipped 1 previous similar message [12593.528166] Lustre: server umount lustre-OST0000 complete [12597.221492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12597.231109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.242118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12601.882719] Lustre: server umount lustre-MDT0000 complete [12604.141224] LustreError: 434695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781250910 with bad export cookie 16541565802563241320 [12604.141941] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12604.154076] LustreError: 434695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12604.325954] Lustre: server umount lustre-MDT0001 complete [12610.172535] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [12611.818201] Key type lgssc unregistered [12612.046343] LNet: 438225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12612.053648] LNetError: 438225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12612.058194] LNet: Removed LNI 192.168.203.113@tcp [12613.087649] LNet: Removed LNI 192.168.203.113@tcp100 [12613.090968] LNetError: 438225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12613.097071] LNetError: 438225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12613.104364] LNet: Removed LNI 192.168.203.113@tcp101 [12614.111156] LNet: Removed LNI 192.168.203.113@tcp102 [12614.115799] LNetError: 438225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12614.122155] LNetError: 438225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12614.128344] LNet: Removed LNI 192.168.203.113@tcp103 [12614.131114] LNet: Removed LNI 192.168.203.113@tcp104 [12614.137314] LNet: Removed LNI 192.168.203.113@tcp105 [12614.143632] LNet: Removed LNI 192.168.203.113@tcp106 [12614.146266] LNet: Removed LNI 192.168.203.113@tcp107 [12614.148728] LNet: Removed LNI 192.168.203.113@tcp108 [12614.151486] LNet: Removed LNI 192.168.203.113@tcp109 [12614.153430] LNet: Removed LNI 192.168.203.113@tcp110 [12614.156636] LNet: Removed LNI 192.168.203.113@tcp111 [12614.158936] LNet: Removed LNI 192.168.203.113@tcp112 [12614.160906] LNet: Removed LNI 192.168.203.113@tcp113 [12614.164067] LNet: Removed LNI 192.168.203.113@tcp114 [12614.166450] LNet: Removed LNI 192.168.203.113@tcp115 [12614.169459] LNet: Removed LNI 192.168.203.113@tcp116 [12614.172249] LNet: Removed LNI 192.168.203.113@tcp117 [12614.175313] LNet: Removed LNI 192.168.203.113@tcp118 [12614.177555] LNet: Removed LNI 192.168.203.113@tcp119 [12614.180506] LNet: Removed LNI 192.168.203.113@tcp120 [12614.183195] LNet: Removed LNI 192.168.203.113@tcp121 [12614.185513] LNet: Removed LNI 192.168.203.113@tcp122 [12614.188099] LNet: Removed LNI 192.168.203.113@tcp123 [12614.190227] LNet: Removed LNI 192.168.203.113@tcp124 [12614.192701] LNet: Removed LNI 192.168.203.113@tcp125 [12614.194980] LNet: Removed LNI 192.168.203.113@tcp126 [12614.197360] LNet: Removed LNI 192.168.203.113@tcp127 [12614.200431] LNet: Removed LNI 192.168.203.113@tcp128 [12614.203535] LNet: Removed LNI 192.168.203.113@tcp129 [12614.206483] LNet: Removed LNI 192.168.203.113@tcp130 [12614.209719] LNet: Removed LNI 192.168.203.113@tcp131 [12614.212386] LNet: Removed LNI 192.168.203.113@tcp132 [12614.216374] LNet: Removed LNI 192.168.203.113@tcp133 [12614.218964] LNet: Removed LNI 192.168.203.113@tcp134 [12614.229953] LNet: Removed LNI 192.168.203.113@tcp135 [12614.747848] Key type .llcrypt unregistered [12614.754252] Key type ._llcrypt unregistered [12627.940997] Key type ._llcrypt registered [12627.944132] Key type .llcrypt registered [12628.008291] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [12636.980453] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [12637.596959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12637.610525] alg: No test for adler32 (adler32-zlib) [12638.519492] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [12638.645911] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [12640.255183] Key type lgssc registered [12640.893927] Lustre: Echo OBD driver; http://www.lustre.org/ [12658.178293] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:56:02 (1781250962) [12665.468472] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [12666.716050] Key type lgssc unregistered [12666.871589] LNet: 442250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12666.878461] LNetError: 442250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12666.891512] LNet: Removed LNI 192.168.203.113@tcp [12667.255552] Key type .llcrypt unregistered [12667.257304] Key type ._llcrypt unregistered [12676.897864] Key type ._llcrypt registered [12676.899487] Key type .llcrypt registered [12676.963707] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [12677.412711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12677.489888] alg: No test for adler32 (adler32-zlib) [12678.397574] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [12678.506249] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [12680.111154] Key type lgssc registered [12680.733600] Lustre: Echo OBD driver; http://www.lustre.org/ [12686.136359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12686.152042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12687.285458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12687.310568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12687.366753] Lustre: lustre-MDT0000: new disk, initializing [12687.433941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12687.447333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12689.774926] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12696.032110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.119633] Lustre: 443422:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12696.159257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12696.162895] Lustre: Skipped 1 previous similar message [12696.206387] Lustre: lustre-MDT0001: new disk, initializing [12696.273688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12696.306931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12696.314790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12698.376401] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12703.023635] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12704.884286] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12708.432624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12708.610615] Lustre: lustre-OST0000: new disk, initializing [12708.616551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12708.625338] Lustre: 445165:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12708.687923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12712.036541] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12716.835596] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12718.609755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12718.617321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12718.699658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12719.119964] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing lnet_if_list [12721.875548] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12721.895422] LNet: Added LNI 192.168.203.113@tcp10 [8/256/0/180] [12725.016853] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12725.035182] LNet: Added LNI 192.168.203.113@tcp11 [8/256/0/180] [12727.203654] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12727.220470] LNet: Added LNI 192.168.203.113@tcp12 [8/256/0/180] [12729.385341] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12729.409097] LNet: Added LNI 192.168.203.113@tcp13 [8/256/0/180] [12731.633803] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12731.654520] LNet: Added LNI 192.168.203.113@tcp14 [8/256/0/180] [12733.940559] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12733.964708] LNet: Added LNI 192.168.203.113@tcp15 [8/256/0/180] [12736.210247] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12736.229858] LNet: Added LNI 192.168.203.113@tcp16 [8/256/0/180] [12738.692751] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12738.712160] LNet: Added LNI 192.168.203.113@tcp17 [8/256/0/180] [12741.276177] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12741.294619] LNet: Added LNI 192.168.203.113@tcp18 [8/256/0/180] [12743.748225] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12743.766831] LNet: Added LNI 192.168.203.113@tcp19 [8/256/0/180] [12746.094524] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12746.115235] LNet: Added LNI 192.168.203.113@tcp20 [8/256/0/180] [12748.527829] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12748.544548] LNet: Added LNI 192.168.203.113@tcp21 [8/256/0/180] [12751.159818] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12751.175726] LNet: Added LNI 192.168.203.113@tcp22 [8/256/0/180] [12753.755423] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12753.773834] LNet: Added LNI 192.168.203.113@tcp23 [8/256/0/180] [12756.868303] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12756.885846] LNet: Added LNI 192.168.203.113@tcp24 [8/256/0/180] [12759.864412] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12759.885589] LNet: Added LNI 192.168.203.113@tcp25 [8/256/0/180] [12763.184990] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12763.209850] LNet: Added LNI 192.168.203.113@tcp26 [8/256/0/180] [12766.289174] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12766.309417] LNet: Added LNI 192.168.203.113@tcp27 [8/256/0/180] [12769.396110] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12769.413705] LNet: Added LNI 192.168.203.113@tcp28 [8/256/0/180] [12772.768590] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12772.788510] LNet: Added LNI 192.168.203.113@tcp29 [8/256/0/180] [12775.991698] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12776.006854] LNet: Added LNI 192.168.203.113@tcp30 [8/256/0/180] [12779.172501] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12779.192411] LNet: Added LNI 192.168.203.113@tcp31 [8/256/0/180] [12782.600989] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12782.621100] LNet: Added LNI 192.168.203.113@tcp32 [8/256/0/180] [12786.062499] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12786.083638] LNet: Added LNI 192.168.203.113@tcp33 [8/256/0/180] [12789.386410] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12789.403490] LNet: Added LNI 192.168.203.113@tcp34 [8/256/0/180] [12792.817084] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12792.833208] LNet: Added LNI 192.168.203.113@tcp35 [8/256/0/180] [12795.981470] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12796.000663] LNet: Added LNI 192.168.203.113@tcp36 [8/256/0/180] [12799.278322] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12799.296985] LNet: Added LNI 192.168.203.113@tcp37 [8/256/0/180] [12802.049023] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12802.063861] LNet: Added LNI 192.168.203.113@tcp38 [8/256/0/180] [12805.041438] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12805.058550] LNet: Added LNI 192.168.203.113@tcp39 [8/256/0/180] [12808.485486] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12808.502360] LNet: Added LNI 192.168.203.113@tcp40 [8/256/0/180] [12812.165584] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12812.185915] LNet: Added LNI 192.168.203.113@tcp41 [8/256/0/180] [12815.625280] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12815.639762] LNet: Added LNI 192.168.203.113@tcp42 [8/256/0/180] [12819.040717] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12819.067228] LNet: Added LNI 192.168.203.113@tcp43 [8/256/0/180] [12822.634681] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12822.656911] LNet: Added LNI 192.168.203.113@tcp44 [8/256/0/180] [12826.796489] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12826.815575] LNet: Added LNI 192.168.203.113@tcp45 [8/256/0/180] [12831.395573] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12831.421735] LNet: Added LNI 192.168.203.113@tcp46 [8/256/0/180] [12835.264990] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12835.286417] LNet: Added LNI 192.168.203.113@tcp47 [8/256/0/180] [12839.561991] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12839.586527] LNet: Added LNI 192.168.203.113@tcp48 [8/256/0/180] [12843.742937] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12843.761398] LNet: Added LNI 192.168.203.113@tcp49 [8/256/0/180] [12848.336735] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12848.362768] LNet: Added LNI 192.168.203.113@tcp50 [8/256/0/180] [12860.386802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12860.391185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12860.398954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12861.921660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12861.922115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12861.931838] Lustre: Skipped 2 previous similar messages [12864.900132] Lustre: server umount lustre-MDT0000 complete [12866.963452] LustreError: 443416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251172 with bad export cookie 8896255658011120074 [12866.966296] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12866.971865] LustreError: 443416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12867.041370] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12867.053111] Lustre: Skipped 1 previous similar message [12867.056970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12867.159122] Lustre: server umount lustre-MDT0001 complete [12877.115260] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [12881.612884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12891.730388] LustreError: 445158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12891.743703] LustreError: 445158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12891.765642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12893.722093] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12897.250205] LustreError: 445159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.876281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12898.104897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12900.353066] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12903.226428] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12905.217438] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12906.472417] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12916.709654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12916.711059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12916.717174] Lustre: Skipped 1 previous similar message [12917.697707] Lustre: server umount lustre-OST0000 complete [12921.824791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12921.827855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12921.835805] Lustre: Skipped 1 previous similar message [12921.847111] Lustre: Skipped 2 previous similar messages [12925.726769] Lustre: server umount lustre-MDT0000 complete [12926.944108] LustreError: 468515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12926.950830] LustreError: 468515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12927.455813] LustreError: 443416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251233 with bad export cookie 8896255658011122300 [12927.459143] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12927.460917] LustreError: 443416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12927.610745] Lustre: server umount lustre-MDT0001 complete [12932.208162] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [12933.593388] Key type lgssc unregistered [12933.833897] LNet: 471045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12933.839436] LNetError: 471045:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12933.844490] LNet: Removed LNI 192.168.203.113@tcp [12933.846945] LNet: Removed LNI 192.168.203.113@tcp10 [12933.849625] LNet: Removed LNI 192.168.203.113@tcp11 [12933.854672] LNet: Removed LNI 192.168.203.113@tcp12 [12933.859433] LNet: Removed LNI 192.168.203.113@tcp13 [12933.861312] LNet: Removed LNI 192.168.203.113@tcp14 [12933.862835] LNet: Removed LNI 192.168.203.113@tcp15 [12933.864659] LNet: Removed LNI 192.168.203.113@tcp16 [12933.866648] LNet: Removed LNI 192.168.203.113@tcp17 [12933.868459] LNet: Removed LNI 192.168.203.113@tcp18 [12933.870587] LNet: Removed LNI 192.168.203.113@tcp19 [12933.872762] LNet: Removed LNI 192.168.203.113@tcp20 [12933.875348] LNet: Removed LNI 192.168.203.113@tcp21 [12933.877456] LNet: Removed LNI 192.168.203.113@tcp22 [12933.879768] LNet: Removed LNI 192.168.203.113@tcp23 [12933.882227] LNet: Removed LNI 192.168.203.113@tcp24 [12933.883844] LNet: Removed LNI 192.168.203.113@tcp25 [12933.886338] LNet: Removed LNI 192.168.203.113@tcp26 [12933.888204] LNet: Removed LNI 192.168.203.113@tcp27 [12933.890255] LNet: Removed LNI 192.168.203.113@tcp28 [12933.892498] LNet: Removed LNI 192.168.203.113@tcp29 [12933.894634] LNet: Removed LNI 192.168.203.113@tcp30 [12933.896830] LNet: Removed LNI 192.168.203.113@tcp31 [12933.898509] LNet: Removed LNI 192.168.203.113@tcp32 [12933.900305] LNet: Removed LNI 192.168.203.113@tcp33 [12933.902370] LNet: Removed LNI 192.168.203.113@tcp34 [12933.904424] LNet: Removed LNI 192.168.203.113@tcp35 [12933.906528] LNet: Removed LNI 192.168.203.113@tcp36 [12933.908666] LNet: Removed LNI 192.168.203.113@tcp37 [12933.911117] LNet: Removed LNI 192.168.203.113@tcp38 [12933.913073] LNet: Removed LNI 192.168.203.113@tcp39 [12933.915237] LNet: Removed LNI 192.168.203.113@tcp40 [12933.917191] LNet: Removed LNI 192.168.203.113@tcp41 [12933.919175] LNet: Removed LNI 192.168.203.113@tcp42 [12933.921329] LNet: Removed LNI 192.168.203.113@tcp43 [12933.923126] LNet: Removed LNI 192.168.203.113@tcp44 [12933.924948] LNet: Removed LNI 192.168.203.113@tcp45 [12933.927610] LNet: Removed LNI 192.168.203.113@tcp46 [12933.929741] LNet: Removed LNI 192.168.203.113@tcp47 [12933.932250] LNet: Removed LNI 192.168.203.113@tcp48 [12933.937259] LNet: Removed LNI 192.168.203.113@tcp49 [12933.945034] LNet: Removed LNI 192.168.203.113@tcp50 [12934.400472] Key type .llcrypt unregistered [12934.404541] Key type ._llcrypt unregistered [12944.713145] Key type ._llcrypt registered [12944.715052] Key type .llcrypt registered [12944.772820] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [12952.372110] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [12952.712480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12952.731933] alg: No test for adler32 (adler32-zlib) [12953.634854] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [12953.750434] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [12955.359202] Key type lgssc registered [12955.888954] Lustre: Echo OBD driver; http://www.lustre.org/ [12974.112594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12974.149035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12975.315854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12975.356639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12975.416781] Lustre: lustre-MDT0000: new disk, initializing [12975.476638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12975.488183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12978.174427] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12985.128825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12985.212817] Lustre: 474196:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12985.257419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12985.260140] Lustre: Skipped 1 previous similar message [12985.310654] Lustre: lustre-MDT0001: new disk, initializing [12985.371162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12985.394917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12985.412264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12988.200837] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [12993.441477] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12996.049531] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13000.874601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13000.993995] Lustre: *** cfs_fail_loc=90f, val=0*** [13001.055542] Lustre: lustre-OST0000: new disk, initializing [13001.058543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13001.063798] Lustre: 475983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13001.111925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13004.398898] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13009.051192] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13009.931635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13009.937075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13010.043643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13015.009256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13015.013723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13015.015457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13015.026423] Lustre: Skipped 1 previous similar message [13019.147631] Lustre: server umount lustre-OST0000 complete [13025.249426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13025.250379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13025.256138] Lustre: Skipped 1 previous similar message [13025.260524] Lustre: Skipped 1 previous similar message [13027.441378] Lustre: server umount lustre-MDT0000 complete [13029.224701] LustreError: 474187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251335 with bad export cookie 8928741535636140421 [13029.225597] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13029.234134] LustreError: 474187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13029.408960] Lustre: server umount lustre-MDT0001 complete [13034.958177] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [13036.456815] Key type lgssc unregistered [13036.615255] LNet: 477755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13036.622714] LNetError: 477755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13036.638634] LNet: Removed LNI 192.168.203.113@tcp [13037.093076] Key type .llcrypt unregistered [13037.097557] Key type ._llcrypt unregistered [13048.260811] Key type ._llcrypt registered [13048.263762] Key type .llcrypt registered [13048.324708] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [13049.019438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13049.032032] alg: No test for adler32 (adler32-zlib) [13049.977977] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [13050.110088] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [13051.711166] Key type lgssc registered [13052.342284] Lustre: Echo OBD driver; http://www.lustre.org/ [13056.547776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13056.566853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13057.808855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13060.009634] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13064.630158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13064.829016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13067.088682] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13070.122309] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13072.084065] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13075.833136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13076.008900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13078.995619] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13081.522612] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13097.243670] loop0: detected capacity change from 0 to 204800000 [13097.335347] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13097.340803] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13097.349335] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13100.380687] loop0: detected capacity change from 0 to 204800000 [13100.472243] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13100.479240] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13100.485617] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13103.253717] loop0: detected capacity change from 0 to 204800000 [13106.176967] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13106.264345] Lustre: fs15246-OST0000: new disk, initializing [13106.268548] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13106.274295] Lustre: 482258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13106.308643] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13109.180203] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13115.049044] loop1: detected capacity change from 0 to 204800000 [13118.360604] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13118.433994] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13118.465656] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13118.509691] Lustre: fs15246-MDT0000: new disk, initializing [13118.609751] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13118.630886] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13118.707244] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13119.779761] Lustre: 483361:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000003fc3a06d x1867777421906048/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781251441 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13120.973592] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13124.834447] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13124.841910] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13124.854041] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13134.309682] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13134.319335] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13138.489570] Lustre: server umount fs15246-MDT0000 complete [13139.427973] LustreError: 478981:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.437628] LustreError: 478981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13142.793199] Lustre: server umount fs15246-OST0000 complete [13148.642783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13148.648167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13148.659319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13149.666758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13149.667779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13153.727727] Lustre: server umount lustre-OST0000 complete [13156.833627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13156.838465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13156.850042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13156.852856] Lustre: Skipped 1 previous similar message [13161.787547] Lustre: server umount lustre-MDT0000 complete [13163.648149] LustreError: 483355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251469 with bad export cookie 1417334465344242579 [13163.651629] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13163.652982] LustreError: 483355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13163.834505] Lustre: server umount lustre-MDT0001 complete [13168.967774] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [13170.400859] Key type lgssc unregistered [13170.559451] LNet: 486418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13170.570185] LNetError: 486418:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13170.586213] LNet: Removed LNI 192.168.203.113@tcp [13170.992439] Key type .llcrypt unregistered [13170.993681] Key type ._llcrypt unregistered [13181.002689] Key type ._llcrypt registered [13181.007641] Key type .llcrypt registered [13181.060751] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [13181.459135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13181.505416] alg: No test for adler32 (adler32-zlib) [13182.415766] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [13182.529356] LNet: Added LNI 192.168.203.113@tcp [8/256/0/180] [13184.143216] Key type lgssc registered [13184.698662] Lustre: Echo OBD driver; http://www.lustre.org/ [13202.651348] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [13207.330529] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [13224.554467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13224.568343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.692774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13225.708701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13225.750955] Lustre: lustre-MDT0000: new disk, initializing [13225.793855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13225.804394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13227.566194] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13232.845237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13232.898601] Lustre: 490805:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13232.925319] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13232.927824] Lustre: Skipped 1 previous similar message [13232.967923] Lustre: lustre-MDT0001: new disk, initializing [13233.009747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13233.031453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13233.037851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13234.926879] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13238.842889] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13240.626789] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13243.887800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13244.024878] Lustre: lustre-OST0000: new disk, initializing [13244.027596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13244.031330] Lustre: 492547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13244.076301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13247.037480] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_default_debug -1 all [13250.569776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13250.575069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13250.631310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13250.926424] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13252.996826] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13253.119964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13255.347148] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13255.473486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13260.770522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13260.774174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13260.787724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13260.791024] Lustre: Skipped 1 previous similar message [13262.561878] Lustre: server umount lustre-OST0000 complete [13265.889646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.890829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13265.899096] Lustre: Skipped 2 previous similar messages [13265.904728] Lustre: Skipped 1 previous similar message [13270.549423] Lustre: server umount lustre-MDT0000 complete [13272.302297] LustreError: 492546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781251578 with bad export cookie 12624037431561246277 [13272.307241] LustreError: MGC192.168.203.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13272.308372] LustreError: 492546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13272.511560] Lustre: server umount lustre-MDT0001 complete [13283.826694] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing set_hostid [13288.224073] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing load_modules_local [13301.919194] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13021 sec ========= 04:06:46 (1781251606) [13302.648655] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:06:46 (1781251606) === [13304.142143] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:06:48 (1781251608) === [13322.173631] Lustre: DEBUG MARKER: oleg313-server.virtnet: executing unload_modules_local [13323.562419] Key type lgssc unregistered [13323.771408] LNet: 498984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13323.777182] LNetError: 498984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13323.789899] LNet: Removed LNI 192.168.203.113@tcp [13324.238732] Key type .llcrypt unregistered [13324.240754] Key type ._llcrypt unregistered