[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466675100 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002400] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012145] LSM: Security Framework initializing [ 0.013056] Yama: becoming mindful. [ 0.015058] SELinux: Initializing. [ 0.017018] *** VALIDATE selinux *** [ 0.026252] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031749] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034015] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035101] *** VALIDATE tmpfs *** [ 0.037061] *** VALIDATE proc *** [ 0.038235] *** VALIDATE cgroup *** [ 0.040006] *** VALIDATE cgroup2 *** [ 0.041268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044026] Spectre V2 : User space: Vulnerable [ 0.045007] Speculative Store Bypass: Vulnerable [ 0.048204] debug: unmapping init [mem 0xffffffffa9059000-0xffffffffa9060fff] [ 0.051181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052604] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053021] ... version: 2 [ 0.054010] ... bit width: 48 [ 0.055011] ... generic registers: 4 [ 0.055955] ... value mask: 0000ffffffffffff [ 0.056015] ... max period: 00007fffffffffff [ 0.057013] ... fixed-purpose events: 3 [ 0.058010] ... event mask: 000000070000000f [ 0.059294] rcu: Hierarchical SRCU implementation. [ 0.061412] smp: Bringing up secondary CPUs ... [ 0.062625] x86: Booting SMP configuration: [ 0.063027] .... node #0, CPUs: #1 #2 #3 [ 0.066337] smp: Brought up 1 node, 4 CPUs [ 0.067977] smpboot: Max logical packages: 1 [ 0.068013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148768] node 0 deferred pages initialised in 77ms [ 0.151448] devtmpfs: initialized [ 0.152215] x86/mm: Memory block size: 128MB [ 0.154853] gcov: version magic: 0x41383552 [ 0.156270] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158238] pinctrl core: initialized pinctrl subsystem [ 0.159138] [ 0.159464] ************************************************************* [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161013] ** ** [ 0.162014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163012] ** ** [ 0.164012] ** This means that this kernel is built to expose internal ** [ 0.165013] ** IOMMU data structures, which may compromise security on ** [ 0.166013] ** your system. ** [ 0.167013] ** ** [ 0.168012] ** If you see this message and you are not debugging the ** [ 0.169012] ** kernel, report this immediately to your vendor! ** [ 0.170012] ** ** [ 0.171024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172014] ************************************************************* [ 0.173611] NET: Registered protocol family 16 [ 0.174384] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177370] cpuidle: using governor menu [ 0.180406] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184482] PCI: Using configuration type 1 for base access [ 0.187132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198056] cryptd: max_cpu_qlen set to 1000 [ 0.201230] ACPI: Added _OSI(Module Device) [ 0.202016] ACPI: Added _OSI(Processor Device) [ 0.204017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212495] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221114] ACPI: Interpreter enabled [ 0.222044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.222987] ACPI: Using IOAPIC for interrupt routing [ 0.223086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233303] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250517] acpiphp: Slot [2] registered [ 0.252135] acpiphp: Slot [5] registered [ 0.254128] acpiphp: Slot [6] registered [ 0.255128] acpiphp: Slot [7] registered [ 0.257138] acpiphp: Slot [8] registered [ 0.259080] acpiphp: Slot [9] registered [ 0.261136] acpiphp: Slot [10] registered [ 0.263137] acpiphp: Slot [3] registered [ 0.265107] acpiphp: Slot [4] registered [ 0.267106] acpiphp: Slot [11] registered [ 0.269133] acpiphp: Slot [12] registered [ 0.271110] acpiphp: Slot [13] registered [ 0.273116] acpiphp: Slot [14] registered [ 0.275121] acpiphp: Slot [15] registered [ 0.277119] acpiphp: Slot [16] registered [ 0.279113] acpiphp: Slot [17] registered [ 0.280101] acpiphp: Slot [18] registered [ 0.282126] acpiphp: Slot [19] registered [ 0.283114] acpiphp: Slot [20] registered [ 0.285141] acpiphp: Slot [21] registered [ 0.286072] acpiphp: Slot [22] registered [ 0.287105] acpiphp: Slot [23] registered [ 0.289080] acpiphp: Slot [24] registered [ 0.290123] acpiphp: Slot [25] registered [ 0.292067] acpiphp: Slot [26] registered [ 0.293099] acpiphp: Slot [27] registered [ 0.295128] acpiphp: Slot [28] registered [ 0.297113] acpiphp: Slot [29] registered [ 0.298127] acpiphp: Slot [30] registered [ 0.300106] acpiphp: Slot [31] registered [ 0.302084] PCI host bridge to bus 0000:00 [ 0.304022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.310022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.313027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.316024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.320086] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326297] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329487] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340871] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346979] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355794] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358938] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364719] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388843] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393726] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.409015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.467015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486936] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.575884] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.593015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.613014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.623287] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.635014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.647014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.665014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.675434] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.681010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.688014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.712011] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.723104] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.731014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.760014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.769090] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.772273] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.776377] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.780450] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.784324] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.790199] iommu: Default domain type: Passthrough [ 0.791461] SCSI subsystem initialized [ 0.792127] ACPI: bus type USB registered [ 0.794105] usbcore: registered new interface driver usbfs [ 0.796073] usbcore: registered new interface driver hub [ 0.798082] usbcore: registered new device driver usb [ 0.799184] pps_core: LinuxPPS API ver. 1 registered [ 0.801011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.804047] PTP clock support registered [ 0.806121] EDAC MC: Ver: 3.0.0 [ 0.808128] PCI: Using ACPI for IRQ routing [ 0.809441] NetLabel: Initializing [ 0.810000] NetLabel: domain hash size = 128 [ 0.811014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.813077] NetLabel: unlabeled traffic allowed by default [ 0.815140] vgaarb: loaded [ 0.816181] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.818010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.824000] clocksource: Switched to clocksource kvm-clock [ 0.919644] VFS: Disk quotas dquot_6.6.0 [ 0.921067] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.922678] *** VALIDATE ramfs *** [ 0.923541] *** VALIDATE hugetlbfs *** [ 0.924534] pnp: PnP ACPI init [ 0.926133] pnp: PnP ACPI: found 6 devices [ 0.942186] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.945834] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.949620] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.953665] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.957676] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.961225] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.965148] NET: Registered protocol family 2 [ 0.968139] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.973993] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.978781] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.984808] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.988429] TCP: Hash tables configured (established 65536 bind 65536) [ 0.990733] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.993720] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.996264] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.999110] NET: Registered protocol family 1 [ 1.002496] RPC: Registered named UNIX socket transport module. [ 1.004477] RPC: Registered udp transport module. [ 1.006052] RPC: Registered tcp transport module. [ 1.007552] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.009689] NET: Registered protocol family 44 [ 1.011137] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.013156] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.015091] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.017285] PCI: CLS 0 bytes, default 64 [ 1.018759] Unpacking initramfs... [ 2.452675] debug: unmapping init [mem 0xffff9d833cc54000-0xffff9d833ffbffff] [ 2.455968] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.458268] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.460968] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.957705] Initialise system trusted keyrings [ 2.959749] Key type blacklist registered [ 2.961876] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.972168] zbud: loaded [ 2.975368] *** VALIDATE nfs *** [ 2.976653] *** VALIDATE nfs4 *** [ 2.978567] pstore: using deflate compression [ 2.982193] Platform Keyring initialized [ 3.090085] NET: Registered protocol family 38 [ 3.093317] Key type asymmetric registered [ 3.095813] Asymmetric key parser 'x509' registered [ 3.097745] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.101032] io scheduler mq-deadline registered [ 3.103033] io scheduler kyber registered [ 3.104238] io scheduler bfq registered [ 3.106263] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.108984] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.111711] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.114358] ACPI: Power Button [PWRF] [ 3.119345] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.126255] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.138094] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.145303] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.165347] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.192604] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.222191] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.227817] Non-volatile memory driver v1.3 [ 3.229523] Linux agpgart interface v0.103 [ 3.265818] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.269374] vda: detected capacity change from 0 to 76791808 [ 3.283763] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.287202] vdb: detected capacity change from 0 to 1073741824 [ 3.307062] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310680] vdc: detected capacity change from 0 to 2621440000 [ 3.328242] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.332248] vdd: detected capacity change from 0 to 2621440000 [ 3.351901] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.355365] vde: detected capacity change from 0 to 4294967296 [ 3.372335] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376196] vdf: detected capacity change from 0 to 4294967296 [ 3.384505] libphy: Fixed MDIO Bus: probed [ 3.389686] usbcore: registered new interface driver usbserial_generic [ 3.391935] usbserial: USB Serial support registered for generic [ 3.394536] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.399662] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.401874] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.404408] mousedev: PS/2 mouse device common for all mice [ 3.407422] rtc_cmos 00:05: RTC can wake from S4 [ 3.411234] rtc_cmos 00:05: registered as rtc0 [ 3.413315] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.416549] intel_pstate: CPU model not supported [ 3.416825] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.424594] hid: raw HID events driver (C) Jiri Kosina [ 3.428674] usbcore: registered new interface driver usbhid [ 3.429331] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.431259] usbhid: USB HID core driver [ 3.431432] drop_monitor: Initializing network drop monitor service [ 3.436640] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.437086] Initializing XFRM netlink socket [ 3.444384] NET: Registered protocol family 10 [ 3.447082] Segment Routing with IPv6 [ 3.448247] NET: Registered protocol family 17 [ 3.449796] mpls_gso: MPLS GSO support [ 3.454629] RAS: Correctable Errors collector initialized. [ 3.456654] AVX version of gcm_enc/dec engaged. [ 3.457901] AES CTR mode by8 optimization enabled [ 3.523947] sched_clock: Marking stable (3523909339, 0)->(4451167986, -927258647) [ 3.527721] registered taskstats version 1 [ 3.530502] Loading compiled-in X.509 certificates [ 3.532280] zswap: loaded using pool lzo/zbud [ 3.556820] Key type big_key registered [ 3.569810] Key type encrypted registered [ 3.571103] ima: No TPM chip found, activating TPM-bypass! [ 3.572537] ima: Allocated hash algorithm: sha1 [ 3.574091] ima: No architecture policies found [ 3.575804] evm: Initialising EVM extended attributes: [ 3.577649] evm: security.selinux [ 3.578884] evm: security.ima [ 3.579894] evm: security.capability [ 3.581247] evm: HMAC attrs: 0x1 [ 3.584460] rtc_cmos 00:05: setting system clock to 2026-09-10 14:00:47 UTC (1789048847) [ 3.590608] debug: unmapping init [mem 0xffffffffaa003000-0xffffffffaa1fffff] [ 3.593223] debug: unmapping init [mem 0xffffffffa8d82000-0xffffffffa9058fff] [ 3.602081] Write protecting the kernel read-only data: 28672k [ 3.605599] debug: unmapping init [mem 0xffffffffa7403000-0xffffffffa75fffff] [ 3.608523] debug: unmapping init [mem 0xffffffffa7d14000-0xffffffffa7dfffff] [ 3.643192] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.651501] systemd[1]: Detected virtualization kvm. [ 3.653624] systemd[1]: Detected architecture x86-64. [ 3.655534] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.684642] systemd[1]: No hostname configured. [ 3.686619] systemd[1]: Set hostname to . [ 3.689392] random: systemd: uninitialized urandom read (16 bytes read) [ 3.692500] systemd[1]: Initializing machine ID from random generator. [ 3.737355] random: ln: uninitialized urandom read (6 bytes read) [ 3.818221] random: systemd: uninitialized urandom read (16 bytes read) [ 3.820786] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.824675] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.827645] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.387303] device-mapper: uevent: version 1.0.3 [ 4.388899] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.225895] virtio_net virtio0 ens2: renamed from eth0 [ 5.243954] random: fast init done [ 5.381116] scsi host0: ata_piix [ 5.400207] scsi host1: ata_piix [ 5.404241] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.406366] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.911984] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.088096] random: crng init done [ 10.089606] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.684462] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.930572] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.189809] SELinux: Disabled at runtime. [ 12.256051] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.265544] systemd[1]: Detected virtualization kvm. [ 12.267040] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.858137] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.861810] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.866523] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.872430] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.875917] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.884445] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.891381] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ 13.046778] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.325463] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.612940] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.692134] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.783164] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.800880] EDAC sbridge: Ver: 1.1.2 [ 15.757368] Key type dns_resolver registered [ 16.070872] NFS: Registering the id_resolver key type [ 16.072838] Key type id_resolver registered [ 16.076366] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-server login: [ 68.098487] libcfs: loading out-of-tree module taints kernel. [ 68.149155] Key type ._llcrypt registered [ 68.152421] Key type .llcrypt registered [ 68.322486] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 86.764890] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 88.668267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.683151] alg: No test for adler32 (adler32-zlib) [ 90.120984] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 91.011146] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 92.927273] Key type lgssc registered [ 95.692487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.897767] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.619979] hrtimer: interrupt took 4050270 ns [ 155.252683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 166.961393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.011887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.254201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.314185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.441505] Lustre: lustre-MDT0000: new disk, initializing [ 168.536066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.556552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.075467] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.565444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.653769] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.680387] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.684563] Lustre: Skipped 1 previous similar message [ 185.750217] Lustre: lustre-MDT0001: new disk, initializing [ 185.797083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.825329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 185.836207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.262332] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.730509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.130775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.536499] Lustre: lustre-OST0000: new disk, initializing [ 204.543133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.549842] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.636925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.261470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.272334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.355800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.116958] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.048280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.256622] Lustre: lustre-OST0001: new disk, initializing [ 226.264101] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.280939] Lustre: 9536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.417098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.494664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.531036] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.612512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.012266] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.836821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.576860] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.916763] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.700944] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing yml_node [ 272.967495] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 275.406416] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 277.918500] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 279.344264] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 10:05:21 EDT 2026 [ 294.980263] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.814673] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.826455] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.835138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.840724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.844284] Lustre: Skipped 1 previous similar message [ 318.618473] Lustre: server umount lustre-MDT0000 complete [ 321.902276] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049165 with bad export cookie 5283171186672334669 [ 321.905516] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.918847] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 324.074240] LustreError: 6534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.075602] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.099893] LustreError: 6534:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 324.124034] Lustre: Skipped 2 previous similar messages [ 324.129703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.132119] Lustre: Skipped 3 previous similar messages [ 326.114473] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.116467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.135810] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 326.150221] Lustre: Skipped 1 previous similar message [ 331.234731] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.236661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.259519] LustreError: 6532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 331.265443] Lustre: Skipped 1 previous similar message [ 336.352032] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.352806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.354185] LustreError: 6533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.401372] Lustre: Skipped 1 previous similar message [ 336.730861] Lustre: server umount lustre-MDT0001 complete [ 340.672523] Lustre: server umount lustre-OST0000 complete [ 344.546572] Lustre: server umount lustre-OST0001 complete [ 350.945219] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 361.213790] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 398.155685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.429267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 398.463504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 398.620847] Lustre: lustre-MDT0000: new disk, initializing [ 398.729087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 398.751221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.325635] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 414.017497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.151864] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 414.242893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 414.257254] Lustre: Skipped 1 previous similar message [ 414.376465] Lustre: lustre-MDT0001: new disk, initializing [ 414.506625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.549127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.561889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.112054] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 428.553913] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.740349] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.467819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.839373] Lustre: lustre-OST0000: new disk, initializing [ 441.848581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.858809] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.949258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.487024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.500192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.602201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 447.623302] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 456.379990] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 460.553879] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 460.789432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 464.560375] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 464.739393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 469.477883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 469.483494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.499687] Lustre: Skipped 1 previous similar message [ 469.508533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.803108] Lustre: server umount lustre-OST0000 complete [ 476.130217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 476.141517] LustreError: Skipped 1 previous similar message [ 476.147375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.159934] Lustre: Skipped 1 previous similar message [ 482.158873] Lustre: server umount lustre-MDT0000 complete [ 484.832719] LustreError: 15092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.845435] LustreError: 15092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 485.372488] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049329 with bad export cookie 5283171186672336202 [ 485.379845] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.383930] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 485.781894] Lustre: server umount lustre-MDT0001 complete [ 490.132141] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:08:51 (1789049331) [ 492.381216] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 494.257462] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:08:56 (1789049336) [ 500.074588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.535112] LustreError: 19469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.596568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.031849] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 514.185741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.558193] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 525.105337] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 530.085092] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.001042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 542.925076] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 550.963791] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 560.097192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.110292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.122385] Lustre: Skipped 1 previous similar message [ 560.128959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.138912] Lustre: Skipped 4 previous similar messages [ 565.978983] Lustre: server umount lustre-OST0000 complete [ 570.860873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.870398] LustreError: Skipped 1 previous similar message [ 570.879850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.888631] Lustre: Skipped 1 previous similar message [ 575.369283] Lustre: server umount lustre-MDT0000 complete [ 576.481437] LustreError: 20521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.506426] LustreError: 20521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 578.462772] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049422 with bad export cookie 5283171186672337084 [ 578.464208] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.476649] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 578.816388] Lustre: server umount lustre-MDT0001 complete [ 588.982273] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 591.583606] Key type lgssc unregistered [ 591.992287] LNet: 23000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.013768] LNetError: 23000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.037328] LNet: Removed LNI 192.168.203.156@tcp [ 593.196280] Key type .llcrypt unregistered [ 593.201781] Key type ._llcrypt unregistered [ 622.990984] Key type ._llcrypt registered [ 622.993954] Key type .llcrypt registered [ 623.117770] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 624.199601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.367850] alg: No test for adler32 (adler32-zlib) [ 625.604497] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 625.853674] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 627.535216] Key type lgssc registered [ 628.594987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 636.793171] loop: module loaded [ 636.860269] loop0: detected capacity change from 0 to 204800000 [ 637.026820] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.038261] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 637.312618] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 642.777918] loop0: detected capacity change from 0 to 204800000 [ 642.980449] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 643.011591] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 643.027103] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 648.037717] loop0: detected capacity change from 0 to 204800000 [ 654.156363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.198193] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.585740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 655.727188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 655.848244] Lustre: lustre-MDT0000: new disk, initializing [ 656.015131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.057819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 660.436704] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 669.324676] loop1: detected capacity change from 0 to 204800000 [ 673.934876] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.199669] Lustre: lustre-OST07c6: new disk, initializing [ 674.203406] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 674.210772] Lustre: Skipped 1 previous similar message [ 674.227240] Lustre: 26186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 674.325954] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 680.237587] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 681.033065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 681.047342] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 681.107085] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 684.176823] Lustre: 25171:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 684.182446] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 684.186632] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 684.192566] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 684.196986] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 684.201325] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 700.593919] Lustre: 26791:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 700.601515] Lustre: 26791:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 700.606891] Lustre: 26791:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 700.613402] Lustre: 26791:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 700.618573] Lustre: 26791:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 700.623773] Lustre: 26791:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.227376] Lustre: 25171:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 705.235917] Lustre: 25171:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 1 previous similar message [ 705.241431] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 705.246483] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.252456] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 705.257659] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.262295] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 705.267020] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.275526] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 705.280264] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.286204] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 705.291511] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.463033] Lustre: 25171:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 708.477123] Lustre: 25171:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 1 previous similar message [ 708.486494] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 708.492313] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.499515] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 708.506560] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.512096] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 708.519035] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.524043] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 708.531967] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 708.540053] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 708.546965] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.981516] Lustre: 25173:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 712.988874] Lustre: 25173:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 1 previous similar message [ 713.000485] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 713.008051] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.016313] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 713.026467] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.035760] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 713.042874] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.053906] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 713.061669] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.068796] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 713.078079] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.774554] Lustre: 26791:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 723.789266] Lustre: 26791:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 5 previous similar messages [ 723.800012] Lustre: 26791:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 723.806455] Lustre: 26791:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 723.812933] Lustre: 26791:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.820562] Lustre: 26791:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 723.829409] Lustre: 26791:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 723.843484] Lustre: 26791:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 723.854201] Lustre: 26791:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.863490] Lustre: 26791:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 723.874123] Lustre: 26791:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.884859] Lustre: 26791:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 772.526602] Lustre: server umount lustre-OST07c6 complete [ 791.799288] Lustre: server umount lustre-MDT0000 complete [ 807.667497] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 809.925926] Key type lgssc unregistered [ 810.202861] LNet: 28742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.210739] LNetError: 28742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.230591] LNet: Removed LNI 192.168.203.156@tcp [ 810.932171] Key type .llcrypt unregistered [ 810.933670] Key type ._llcrypt unregistered [ 826.828117] Key type ._llcrypt registered [ 826.830335] Key type .llcrypt registered [ 826.908862] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 827.749665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.830091] alg: No test for adler32 (adler32-zlib) [ 828.835783] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 829.003562] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 830.663363] Key type lgssc registered [ 831.740685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 839.320314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 839.354109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.833973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.094193] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 850.404035] LustreError: 29991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.417893] LustreError: 29991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 853.056773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.400612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.570813] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 863.747804] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 869.296584] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 875.303095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.708130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.860281] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 885.933360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 887.301256] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 901.600806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.607292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.609295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 901.615696] Lustre: Skipped 1 previous similar message [ 901.633609] Lustre: Skipped 1 previous similar message [ 906.722288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.738443] Lustre: Skipped 1 previous similar message [ 907.217542] Lustre: server umount lustre-OST0000 complete [ 911.841906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 911.843759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 911.852539] Lustre: Skipped 1 previous similar message [ 916.747319] Lustre: server umount lustre-MDT0000 complete [ 916.964935] LustreError: 29991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.982962] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049763 with bad export cookie 17082424071308586445 [ 919.990886] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 920.013292] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 920.493483] Lustre: server umount lustre-MDT0001 complete [ 930.797626] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 933.463352] Key type lgssc unregistered [ 933.763740] LNet: 33723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.772422] LNetError: 33723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.795661] LNet: Removed LNI 192.168.203.156@tcp [ 934.633475] Key type .llcrypt unregistered [ 934.635868] Key type ._llcrypt unregistered [ 956.745995] Key type ._llcrypt registered [ 956.751385] Key type .llcrypt registered [ 956.899817] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 957.661040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 957.767549] alg: No test for adler32 (adler32-zlib) [ 958.994264] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 959.236838] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 960.919254] Key type lgssc registered [ 961.949509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 970.115667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 970.147801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.757277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.747490] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 981.984995] LustreError: 34990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.001677] LustreError: 34990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 985.512619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.949369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 990.284908] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 998.090719] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.022246] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1010.879139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1011.209225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.331120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1017.618522] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1026.258281] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1040.864832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.874645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.889317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.915058] Lustre: Skipped 1 previous similar message [ 1040.933808] Lustre: Skipped 1 previous similar message [ 1045.984354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.814506] Lustre: server umount lustre-OST0000 complete [ 1051.106150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.129092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.137114] Lustre: Skipped 2 previous similar messages [ 1056.225673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.239237] Lustre: Skipped 1 previous similar message [ 1056.547912] Lustre: server umount lustre-MDT0000 complete [ 1060.733268] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789049904 with bad export cookie 17750643775794194807 [ 1060.744681] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1060.749686] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1061.060056] Lustre: server umount lustre-MDT0001 complete [ 1070.885673] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1073.387565] Key type lgssc unregistered [ 1073.627381] LNet: 38543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.632883] LNetError: 38543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.673500] LNet: Removed LNI 192.168.203.156@tcp [ 1075.702699] Key type .llcrypt unregistered [ 1075.711761] Key type ._llcrypt unregistered [ 1096.461509] Key type ._llcrypt registered [ 1096.463801] Key type .llcrypt registered [ 1096.561949] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1097.425493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1097.486083] alg: No test for adler32 (adler32-zlib) [ 1098.620767] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1098.944794] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1100.673447] Key type lgssc registered [ 1101.984347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.640689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.656578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.040260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.251852] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1120.233715] LustreError: 39938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.257105] LustreError: 39938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1122.831864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.091713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1126.349803] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1132.238663] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1136.771956] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.231438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1142.554137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.672745] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1155.618242] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1155.911471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1161.513326] Lustre: server umount lustre-MDT0000 complete [ 1162.581790] LustreError: 39935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.257438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1164.268939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.290936] LustreError: 41033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.391466] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050009 with bad export cookie 863629755948799232 [ 1165.405527] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.414951] LustreError: 39919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1166.304969] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.322042] LustreError: 39935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.323927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1167.709190] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [ 1170.337256] LustreError: 42377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.340923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1170.357510] LustreError: 42377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1171.747661] Lustre: server umount lustre-MDT0001 complete [ 1192.387358] Lustre: server umount lustre-OST0000 complete [ 1211.866774] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1214.946171] Key type lgssc unregistered [ 1215.328345] LNet: 43797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.338353] LNetError: 43797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.354161] LNet: Removed LNI 192.168.203.156@tcp [ 1216.510482] Key type .llcrypt unregistered [ 1216.512816] Key type ._llcrypt unregistered [ 1239.894628] Key type ._llcrypt registered [ 1239.897985] Key type .llcrypt registered [ 1239.977418] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1240.938076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1240.985575] alg: No test for adler32 (adler32-zlib) [ 1241.974276] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1242.233681] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1243.887795] Key type lgssc registered [ 1244.800624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1251.406704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1251.421464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.795457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.129945] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1263.072773] LustreError: 45208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.112480] LustreError: 45208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1264.723344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.084725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1269.052947] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1275.764939] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1280.877912] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1286.487234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1286.877591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.951267] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1297.136681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1299.637573] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1307.484834] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 1307.626732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.627765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1311.969540] Lustre: server umount lustre-OST0000 complete [ 1331.681208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1331.695795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.709924] Lustre: Skipped 1 previous similar message [ 1331.715502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.720066] Lustre: Skipped 2 previous similar messages [ 1333.220497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1333.230143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.317154] Lustre: server umount lustre-MDT0000 complete [ 1338.339252] LustreError: 45207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.361460] LustreError: 45207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1338.453728] LustreError: 45188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050182 with bad export cookie 12594671294480176849 [ 1338.465386] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1345.062490] Lustre: server umount lustre-MDT0001 complete [ 1355.660838] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1358.429361] Key type lgssc unregistered [ 1358.697890] LNet: 48913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.708850] LNetError: 48913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.733322] LNet: Removed LNI 192.168.203.156@tcp [ 1359.709174] Key type .llcrypt unregistered [ 1359.711649] Key type ._llcrypt unregistered [ 1381.099175] Key type ._llcrypt registered [ 1381.101818] Key type .llcrypt registered [ 1381.220534] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1382.338243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.477487] alg: No test for adler32 (adler32-zlib) [ 1383.548609] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1383.769695] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1385.409305] Key type lgssc registered [ 1386.497719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1395.419734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1395.445362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.009176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1401.635779] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1406.445732] LustreError: 50322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.470415] LustreError: 50322:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1410.724972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.085995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.464424] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1423.036519] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1428.899361] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1434.909626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1435.275824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.198870] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1441.466803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1448.832609] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1453.777493] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1456.765559] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1469.435584] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1010h25m10s [ 1471.564318] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1475.944704] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 1477.044962] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 1477.600636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.603712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1477.618050] Lustre: Skipped 2 previous similar messages [ 1478.702943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.709688] Lustre: Skipped 3 previous similar messages [ 1480.606335] Lustre: server umount lustre-MDT0000 complete [ 1482.103746] LustreError: 50317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.360466] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1484.770605] LustreError: 53008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.792906] LustreError: 53008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1488.196218] LustreError: 52227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.057936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.302723] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.659791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.735642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1493.689807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1497.541400] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1501.271511] Lustre: DEBUG MARKER: Wait for df (36711) ... [ 1502.913717] Lustre: DEBUG MARKER: done [ 1508.321283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1508.328412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.331399] LustreError: Skipped 1 previous similar message [ 1508.335873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1508.345511] Lustre: Skipped 1 previous similar message [ 1513.692533] Lustre: server umount lustre-OST0000 complete [ 1519.586364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1519.593175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1519.594502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1519.594509] Lustre: Skipped 3 previous similar messages [ 1519.615835] Lustre: Skipped 1 previous similar message [ 1523.910676] Lustre: server umount lustre-MDT0000 complete [ 1526.757755] LustreError: 53008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.785868] LustreError: 53008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1527.601955] LustreError: 50301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050371 with bad export cookie 14254893444145473674 [ 1527.617167] LustreError: 50301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1527.618247] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.850843] Lustre: server umount lustre-MDT0001 complete [ 1538.756045] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1541.639333] Key type lgssc unregistered [ 1541.953163] LNet: 55243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.968483] LNetError: 55243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.992860] LNet: Removed LNI 192.168.203.156@tcp [ 1542.712423] Key type .llcrypt unregistered [ 1542.716435] Key type ._llcrypt unregistered [ 1578.424642] Key type ._llcrypt registered [ 1578.427476] Key type .llcrypt registered [ 1578.513300] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1579.870257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1579.921469] alg: No test for adler32 (adler32-zlib) [ 1580.914684] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1581.164247] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1582.847202] Key type lgssc registered [ 1583.895825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1591.088985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1591.133702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1592.445792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1592.463577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1592.645647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.927341] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1604.680278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.867324] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1604.914264] Lustre: 57447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1605.122271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.394809] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1617.487331] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1623.922976] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1630.980389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1631.183679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1631.516300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.787662] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1638.084811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1645.089029] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.911816] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1010h28m14s [ 1653.435852] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1659.847944] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1662.075916] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1662.138789] LustreError: 57454:0:(mdt_handler.c:2357:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1682.285727] Lustre: lustre-MDT0000: Client 0c5ef080-228c-4c8f-b70a-67dc0a171a42 (at 192.168.203.56@tcp) reconnecting [ 1702.239158] LustreError: 57454:0:(mdt_handler.c:2357:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1704.093213] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1706.231503] Lustre: DEBUG MARKER: done [ 1715.172260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1715.176207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.178839] LustreError: Skipped 1 previous similar message [ 1715.180706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1715.209575] Lustre: Skipped 1 previous similar message [ 1715.219166] Lustre: Skipped 1 previous similar message [ 1717.174354] Lustre: server umount lustre-OST0000 complete [ 1722.849225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1722.861164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.900041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.937891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.947858] Lustre: Skipped 2 previous similar messages [ 1727.020193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.611706] Lustre: server umount lustre-MDT0000 complete [ 1732.475192] LustreError: 57438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050576 with bad export cookie 12268538273911654410 [ 1732.489201] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.490120] LustreError: 57438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1732.895937] Lustre: server umount lustre-MDT0001 complete [ 1744.554449] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 1747.801668] Key type lgssc unregistered [ 1748.190906] LNet: 61776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.201264] LNetError: 61776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.224344] LNet: Removed LNI 192.168.203.156@tcp [ 1749.379201] Key type .llcrypt unregistered [ 1749.381506] Key type ._llcrypt unregistered [ 1791.536921] Key type ._llcrypt registered [ 1791.539650] Key type .llcrypt registered [ 1791.653564] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 1792.709947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.861269] alg: No test for adler32 (adler32-zlib) [ 1794.108502] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 1794.518823] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 1796.231355] Key type lgssc registered [ 1797.428403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1801.612576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1801.641338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.986108] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1803.039916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1803.283628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.500054] Lustre: Failing over lustre-MDT0000 [ 1804.976475] Lustre: server umount lustre-MDT0000 complete [ 1813.339553] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:30:55 (1789050655) [ 1819.259556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.751922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.514727] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1832.867792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1832.990824] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1833.027542] Lustre: 64572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1833.295296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.417363] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1845.764502] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.395156] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1858.379755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.558866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1858.779792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.494412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1866.691957] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1874.545582] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1879.704538] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1889.771680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1889.779551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.805794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.243563] Lustre: server umount lustre-MDT0000 complete [ 1896.049380] LustreError: 64564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050739 with bad export cookie 8672767753675316612 [ 1896.055688] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1896.056101] LustreError: 64564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1896.515824] Lustre: server umount lustre-MDT0001 complete [ 1899.878699] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1904.379831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1912.489774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.897611] Lustre: 63495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050744/real 1789050744] req@0000000058c1fb39 x1875955561832704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789050760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1916.923639] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1916.942165] Lustre: Skipped 2 previous similar messages [ 1917.919177] Lustre: 63494:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050745/real 1789050745] req@00000000b57f44c9 x1875955561832832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789050761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1921.572030] LustreError: 66420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.596178] LustreError: 66420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1921.685543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.804200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1922.659616] Lustre: 63492:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050749/real 1789050749] req@00000000b90f4c7a x1875955561832960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789050765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1926.585823] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1926.799164] Lustre: 63492:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050754/real 1789050754] req@00000000a328711a x1875955561833216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789050770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1926.833442] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1926.873147] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1932.269722] LustreError: 68339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1932.299304] LustreError: 68339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1933.285643] Lustre: 63495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050759/real 1789050759] req@00000000b009aa7f x1875955561833344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789050775 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1935.273215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.723350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.347256] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1940.979659] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1945.133547] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1947.082731] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1952.003240] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.213955] Lustre: server umount lustre-MDT0000 complete [ 1960.421857] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.426474] LustreError: 66421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1960.435090] Lustre: Skipped 1 previous similar message [ 1961.245478] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050805 with bad export cookie 8672767753675318887 [ 1961.249738] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.258324] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1963.651296] Lustre: server umount lustre-MDT0001 complete [ 1969.867935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1978.649550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.919738] Lustre: 63494:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789050809/real 1789050809] req@00000000ac74e1c8 x1875955561857536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789050825 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.940788] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.957273] Lustre: Skipped 1 previous similar message [ 1987.109107] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x785bdc525947abca [ 1987.132243] Lustre: MGC192.168.203.156@tcp: Connection restored to 0@lo (at 0@lo) [ 1987.489388] LustreError: 69073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.513820] LustreError: 69073:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1987.574763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.667940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1992.351460] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 1996.778091] LustreError: 71023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.803143] LustreError: 71023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2000.871930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2001.113929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.043035] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2006.513325] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2013.702507] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2019.847798] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2026.980803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2026.983318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.990661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.990671] Lustre: Skipped 3 previous similar messages [ 2027.014938] Lustre: Skipped 2 previous similar messages [ 2032.119699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.122516] Lustre: Skipped 2 previous similar messages [ 2032.857068] Lustre: server umount lustre-MDT0000 complete [ 2036.747348] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050880 with bad export cookie 8672767753675320266 [ 2036.753955] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.769193] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2037.175970] Lustre: server umount lustre-MDT0001 complete [ 2044.780884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2048.242207] Lustre: DEBUG MARKER: files compared the same [ 2060.692291] Lustre: server umount lustre-OST0000 complete [ 2077.167780] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2079.902483] Key type lgssc unregistered [ 2080.174868] LNet: 74298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.194225] LNetError: 74298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.220903] LNet: Removed LNI 192.168.203.156@tcp [ 2081.055331] Key type .llcrypt unregistered [ 2081.060143] Key type ._llcrypt unregistered [ 2107.351282] Key type ._llcrypt registered [ 2107.353723] Key type .llcrypt registered [ 2107.446290] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2108.359844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.385139] alg: No test for adler32 (adler32-zlib) [ 2109.628826] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2109.859533] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2111.511299] Key type lgssc registered [ 2112.743418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2119.944813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2119.973295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.439821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.319885] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all [ 2131.948018] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.975857] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2133.544548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.020379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.202987] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all [ 2146.312960] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2154.077867] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2161.629218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.955873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.099665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2168.133399] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug +malloc all [ 2175.240926] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2187.752219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2187.760207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.767855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.771536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.772304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2188.806454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.797849] Lustre: server umount lustre-OST0000 complete [ 2199.020638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.034725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.040349] Lustre: Skipped 2 previous similar messages [ 2203.970989] Lustre: server umount lustre-MDT0000 complete [ 2204.129672] LustreError: 77709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.152922] LustreError: 77709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2208.223739] LustreError: 75688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051052 with bad export cookie 17871052492737203520 [ 2208.235563] LustreError: 75709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.243816] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2208.243873] LustreError: 75688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2208.636056] Lustre: server umount lustre-MDT0001 complete [ 2218.674805] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2221.251472] Key type lgssc unregistered [ 2221.524870] LNet: 79252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.531292] LNetError: 79252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.578106] LNet: Removed LNI 192.168.203.156@tcp [ 2222.408343] Key type .llcrypt unregistered [ 2222.414283] Key type ._llcrypt unregistered [ 2244.939359] Key type ._llcrypt registered [ 2244.942224] Key type .llcrypt registered [ 2245.038203] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2246.060676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2246.223783] alg: No test for adler32 (adler32-zlib) [ 2247.380826] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2247.635975] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2249.351246] Key type lgssc registered [ 2250.451766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.860695] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.886205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.379134] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2278.495325] LustreError: 80639:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 4: rc = -110 [ 2305.122471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.678739] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2318.632584] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2325.354398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.512418] Lustre: *** cfs_fail_loc=706, val=0*** [ 2325.830399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.920324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2330.214730] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2331.111992] LustreError: 80665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.138036] LustreError: 80665:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2336.236989] LustreError: 80664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.532366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.837978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.933509] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2350.225787] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2354.946862] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2356.704550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2356.722266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.738340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2359.264563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2359.269783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.279018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.291970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.739958] Lustre: server umount lustre-OST0000 complete [ 2369.505260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2369.514860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.534962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.539238] Lustre: Skipped 1 previous similar message [ 2372.615842] Lustre: server umount lustre-MDT0000 complete [ 2375.654976] LustreError: 81726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.677948] LustreError: 81726:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2376.425667] LustreError: 80647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051220 with bad export cookie 13359989659412224073 [ 2376.435734] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2376.446138] LustreError: 80647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2376.777533] Lustre: server umount lustre-MDT0001 complete [ 2387.405711] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2390.432898] Key type lgssc unregistered [ 2390.735418] LNet: 84254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.746608] LNetError: 84254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.776859] LNet: Removed LNI 192.168.203.156@tcp [ 2391.835880] Key type .llcrypt unregistered [ 2391.839538] Key type ._llcrypt unregistered [ 2415.502518] Key type ._llcrypt registered [ 2415.504703] Key type .llcrypt registered [ 2415.615952] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2416.659257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2416.704911] alg: No test for adler32 (adler32-zlib) [ 2417.898568] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2418.213933] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2419.959222] Key type lgssc registered [ 2421.167928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2429.062340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2429.082434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.826714] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2443.904900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.466390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.949146] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2449.889091] LustreError: 86354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.916934] LustreError: 86354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2455.010151] LustreError: 86359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2455.026816] LustreError: 86359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2457.728149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.092392] LustreError: 86354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.114479] LustreError: 86354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2458.116400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.202486] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.222925] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2464.541736] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2468.322674] LustreError: 86355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.349673] LustreError: 86355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2473.364380] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2473.385201] Lustre: MGS: Client f9245089-ddb3-4eff-b15c-9b99e2085a7c (at 0@lo) reconnecting [ 2473.402282] LustreError: 86369:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2473.629122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.718561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2478.145684] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2494.438578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.439196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2494.441219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2494.454976] Lustre: Skipped 1 previous similar message [ 2494.483235] Lustre: Skipped 1 previous similar message [ 2499.436727] Lustre: server umount lustre-OST0000 complete [ 2504.673074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2504.674466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.690855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2504.703339] Lustre: Skipped 1 previous similar message [ 2504.722566] Lustre: Skipped 1 previous similar message [ 2509.635425] Lustre: server umount lustre-MDT0000 complete [ 2509.805647] LustreError: 86359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.831639] LustreError: 86359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2513.942312] Lustre: server umount lustre-MDT0001 complete [ 2518.102283] Lustre: server umount MGS complete [ 2528.111713] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:42:50 (1789051370) [ 2555.666231] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 2566.642199] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2604.282842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.780242] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2620.488279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2620.872596] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2620.928692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2621.052778] Lustre: lustre-MDT0001: new disk, initializing [ 2621.232493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.288774] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2621.301405] Lustre: 93543:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2621.315274] LustreError: 93543:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2626.281515] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2636.624115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.993388] Lustre: lustre-OST0000: new disk, initializing [ 2637.006729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2637.017679] Lustre: 94311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2637.121698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2644.064281] Lustre: 94606:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004aed7485 x1875956216143488/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789051503 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2644.289676] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2654.525168] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2661.323603] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2661.336263] Lustre: MGS: Client 86cdaa82-53db-4759-84f5-aa869f45f6d3 (at 0@lo) reconnecting [ 2661.363225] LustreError: 93529:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2661.407163] Lustre: 95357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2661.455558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2661.548779] Lustre: lustre-MDT0000: new disk, initializing [ 2661.655160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.694181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2661.731088] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2662.761115] Lustre: 95381:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000015e1944 x1875956216150400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789051522 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2665.773863] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2667.935865] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2670.578524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2670.588272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2671.848147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2671.864761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2672.003453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2696.160175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2696.167328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.180279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.256087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2700.261673] Lustre: Skipped 2 previous similar messages [ 2701.638641] Lustre: server umount lustre-OST0000 complete [ 2705.377338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2705.387752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.389759] LustreError: Skipped 1 previous similar message [ 2705.395035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.432554] Lustre: Skipped 2 previous similar messages [ 2711.621807] Lustre: server umount lustre-MDT0000 complete [ 2721.427760] Lustre: server umount lustre-MDT0001 complete [ 2725.076728] Lustre: server umount MGS complete [ 2739.246839] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:46:21 (1789051581) [ 2755.792502] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2758.692519] Key type lgssc unregistered [ 2759.047756] LNet: 97875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2759.063868] LNetError: 97875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2759.091886] LNet: Removed LNI 192.168.203.156@tcp [ 2760.041310] Key type .llcrypt unregistered [ 2760.043548] Key type ._llcrypt unregistered [ 2777.664719] Key type ._llcrypt registered [ 2777.667409] Key type .llcrypt registered [ 2777.801818] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2779.376725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2779.405411] alg: No test for adler32 (adler32-zlib) [ 2780.439279] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2780.766726] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2782.464546] Key type lgssc registered [ 2784.199901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2793.155694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2793.180743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.821321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.350853] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2804.203989] LustreError: 99096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.227187] LustreError: 99096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2807.881749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.178098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.253079] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2819.199199] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2824.794274] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2831.410383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.839789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.258159] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2840.072176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2846.323900] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2855.392796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.400735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.404659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.421010] Lustre: Skipped 1 previous similar message [ 2855.433504] Lustre: Skipped 1 previous similar message [ 2860.527154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.532196] Lustre: Skipped 1 previous similar message [ 2860.819649] Lustre: server umount lustre-OST0000 complete [ 2865.667308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.674220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2865.698196] Lustre: Skipped 1 previous similar message [ 2865.707719] Lustre: Skipped 1 previous similar message [ 2870.756408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.770751] Lustre: Skipped 1 previous similar message [ 2871.135567] Lustre: server umount lustre-MDT0000 complete [ 2875.195357] LustreError: 99077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051719 with bad export cookie 15005586519830453806 [ 2875.221949] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.224181] LustreError: 99077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2875.510816] Lustre: server umount lustre-MDT0001 complete [ 2889.307559] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 2893.382653] Key type lgssc unregistered [ 2893.663687] LNet: 102630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.672758] LNetError: 102630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.703732] LNet: Removed LNI 192.168.203.156@tcp [ 2894.505755] Key type .llcrypt unregistered [ 2894.507302] Key type ._llcrypt unregistered [ 2912.191771] Key type ._llcrypt registered [ 2912.194690] Key type .llcrypt registered [ 2912.315858] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 2913.593793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2913.638915] alg: No test for adler32 (adler32-zlib) [ 2914.666319] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2914.937268] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 2916.655680] Key type lgssc registered [ 2917.604079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2923.201505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2923.206358] LustreError: 103706:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2923.221363] LustreError: 103710:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2923.226293] LustreError: 103706:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 2923.233206] LustreError: 103706:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2923.242362] LustreError: 103706:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2923.243818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2924.637684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.127198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2932.593728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2937.582746] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2943.775894] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing lsmod [ 2946.733393] LustreError: 104911:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2946.782354] LustreError: 104918:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2946.787981] LustreError: 104911:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4953 [ 2946.794766] LustreError: 104911:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2946.803187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.805388] LustreError: 104911:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2947.163241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2949.238805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2953.186679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2953.207146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.228200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.730708] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.731654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.751386] Lustre: Skipped 1 previous similar message [ 2954.760638] Lustre: Skipped 1 previous similar message [ 2956.440704] Lustre: server umount lustre-MDT0000 complete [ 2957.872198] LustreError: 103744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.911222] LustreError: 103744:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2960.543089] LustreError: 103727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051804 with bad export cookie 7270534120059755159 [ 2960.544831] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.558110] LustreError: 103727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2960.589617] Lustre: 103740:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2962.015346] LustreError: 104180:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bb87ee72 x1875956736217600/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2962.195155] Lustre: server umount lustre-MDT0001 complete [ 2977.026276] Lustre: server umount lustre-OST0000 complete [ 2985.263119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.800634] LustreError: 105981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.930215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.632216] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 2991.078394] LustreError: 105982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.182115] LustreError: 105981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.404635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.740278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3003.019557] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3009.794137] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3015.399279] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3023.289446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3023.559651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.772897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3031.060902] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3039.222144] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3051.489118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3051.493958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3051.500307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3051.502348] Lustre: Skipped 1 previous similar message [ 3054.568994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.577979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3054.578528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.619932] Lustre: Skipped 1 previous similar message [ 3056.669347] Lustre: server umount lustre-OST0000 complete [ 3064.801590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.808658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.834583] Lustre: Skipped 1 previous similar message [ 3068.383437] Lustre: server umount lustre-MDT0000 complete [ 3069.928121] LustreError: 107417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.946505] LustreError: 107417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3072.645357] LustreError: 106716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051916 with bad export cookie 7270534120059756083 [ 3072.658775] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.663386] LustreError: 106716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3073.118355] Lustre: server umount lustre-MDT0001 complete [ 3085.403068] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3088.468963] Key type lgssc unregistered [ 3088.752488] LNet: 109514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3088.766376] LNetError: 109514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3089.838516] LNet: Removed LNI 192.168.203.156@tcp [ 3090.898722] Key type .llcrypt unregistered [ 3090.904178] Key type ._llcrypt unregistered [ 3117.574287] Key type ._llcrypt registered [ 3117.583337] Key type .llcrypt registered [ 3117.707751] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3119.312521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3119.353029] alg: No test for adler32 (adler32-zlib) [ 3120.374737] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3120.641551] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3122.360382] Key type lgssc registered [ 3123.645239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3131.847345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3131.869501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3133.440300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3138.706912] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3143.656224] LustreError: 110930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.675169] LustreError: 110930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3148.787897] LustreError: 110931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.264558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.770856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3155.639157] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3164.314666] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3168.978841] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3175.213295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.397484] LustreError: 112752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.607943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.677199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3182.262053] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3190.270596] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3196.725812] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3201.675473] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3209.699304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3209.735297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.774266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3209.786364] Lustre: Skipped 1 previous similar message [ 3213.774820] Lustre: server umount lustre-OST0000 complete [ 3219.940286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.957857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.968603] Lustre: Skipped 2 previous similar messages [ 3219.981232] Lustre: Skipped 1 previous similar message [ 3224.603828] Lustre: server umount lustre-MDT0000 complete [ 3225.056924] LustreError: 110925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.084845] LustreError: 110925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3228.539410] LustreError: 110912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052072 with bad export cookie 4206467009085579627 [ 3228.546659] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3228.549692] LustreError: 110912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3228.871357] Lustre: server umount lustre-MDT0001 complete [ 3239.136141] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3242.122694] Key type lgssc unregistered [ 3242.447635] LNet: 114569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.456737] LNetError: 114569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.475824] LNet: Removed LNI 192.168.203.156@tcp [ 3243.370082] Key type .llcrypt unregistered [ 3243.372180] Key type ._llcrypt unregistered [ 3261.262623] Key type ._llcrypt registered [ 3261.264673] Key type .llcrypt registered [ 3261.373837] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3262.209992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3262.228309] alg: No test for adler32 (adler32-zlib) [ 3263.313993] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3263.562967] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3265.247169] Key type lgssc registered [ 3266.449932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3274.366358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3274.400325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.952105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.909980] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3285.479294] LustreError: 115788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.521891] LustreError: 115788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3289.596325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.979464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3294.812436] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3302.999575] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3309.989109] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3318.514729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3318.804632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.905460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3325.578923] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3334.606884] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3344.353669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3344.367909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.375547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3344.387228] Lustre: Skipped 1 previous similar message [ 3344.403615] Lustre: Skipped 1 previous similar message [ 3349.400796] Lustre: server umount lustre-OST0000 complete [ 3354.596786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.631699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3354.640170] Lustre: Skipped 1 previous similar message [ 3359.716834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.730542] Lustre: Skipped 1 previous similar message [ 3360.045973] Lustre: server umount lustre-MDT0000 complete [ 3364.089874] LustreError: 115767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052208 with bad export cookie 14139303098632834726 [ 3364.097077] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.103235] LustreError: 115767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3364.470347] Lustre: server umount lustre-MDT0001 complete [ 3376.305611] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3379.327687] Key type lgssc unregistered [ 3379.644435] LNet: 119318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3379.669327] LNetError: 119318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3380.709853] LNet: Removed LNI 192.168.203.156@tcp [ 3381.737784] Key type .llcrypt unregistered [ 3381.741542] Key type ._llcrypt unregistered [ 3406.692296] Key type ._llcrypt registered [ 3406.694728] Key type .llcrypt registered [ 3406.792250] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3408.230479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3408.311135] alg: No test for adler32 (adler32-zlib) [ 3409.512240] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3409.856597] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3411.607213] Key type lgssc registered [ 3413.085569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3422.080382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3422.096590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.871718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.929595] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3433.446326] LustreError: 120734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.474904] LustreError: 120734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3438.496204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.566314] LustreError: 120735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.913329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3443.937717] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3452.208773] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3458.904289] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3467.387364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.984664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3469.004987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3475.459296] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3484.488954] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3493.131046] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3497.921452] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3497.927662] Lustre: Skipped 2 previous similar messages [ 3501.269039] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3501.369685] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3503.876945] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3503.883351] Lustre: Skipped 2 previous similar messages [ 3509.393477] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3509.414094] Lustre: Skipped 1 previous similar message [ 3515.059644] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3515.066233] Lustre: Skipped 2 previous similar messages [ 3519.242053] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.56@tcp 192.168.0.[2,10]@tcp [ 3524.673565] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.56/24@tcp in log lustre-MDT0000 [ 3524.690075] Lustre: Skipped 2 previous similar messages [ 3534.471749] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.56/24@tcp [ 3534.475825] Lustre: Skipped 1 previous similar message [ 3539.943711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.944603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3539.945171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3539.992962] Lustre: Skipped 1 previous similar message [ 3545.077830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3545.088797] Lustre: Skipped 2 previous similar messages [ 3546.018732] Lustre: server umount lustre-OST0000 complete [ 3550.176420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.190209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3550.196316] Lustre: Skipped 1 previous similar message [ 3555.296302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3555.311671] Lustre: Skipped 1 previous similar message [ 3556.549272] Lustre: server umount lustre-MDT0000 complete [ 3560.419227] LustreError: 120729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.435834] LustreError: 120729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3560.885612] LustreError: 122578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052404 with bad export cookie 9943345307961074734 [ 3560.892290] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.908197] LustreError: 122578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3561.388545] Lustre: server umount lustre-MDT0001 complete [ 3572.802093] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3576.532189] Key type lgssc unregistered [ 3577.037077] LNet: 125727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.043869] LNetError: 125727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3577.075465] LNet: Removed LNI 192.168.203.156@tcp [ 3578.248601] Key type .llcrypt unregistered [ 3578.250871] Key type ._llcrypt unregistered [ 3609.362833] Key type ._llcrypt registered [ 3609.365933] Key type .llcrypt registered [ 3609.520296] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3610.518944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3610.594618] alg: No test for adler32 (adler32-zlib) [ 3611.721869] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3611.995232] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3613.671212] Key type lgssc registered [ 3615.345901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.062269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.074375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.504253] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3626.543425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.562415] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.56/24@tcp [ 3631.491648] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3636.712662] LustreError: 127299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.737378] LustreError: 127299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3640.244504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.712741] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3640.762033] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.56/24@tcp [ 3640.838345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3646.561567] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3654.731459] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3660.558578] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3668.220053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3668.651791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.760977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3675.578968] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3684.895364] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3697.636787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.651503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.667715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.734669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.735984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.738692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.767947] Lustre: Skipped 1 previous similar message [ 3703.836818] Lustre: server umount lustre-OST0000 complete [ 3707.360336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3707.369224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.386068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.971654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3711.982982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.688575] Lustre: server umount lustre-MDT0000 complete [ 3717.089323] LustreError: 127299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3717.133058] LustreError: 127299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3717.675965] LustreError: 127281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052561 with bad export cookie 10258734965905493134 [ 3717.691407] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3718.151595] Lustre: server umount lustre-MDT0001 complete [ 3729.094941] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 3731.740421] Key type lgssc unregistered [ 3731.976755] LNet: 130888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3731.986993] LNetError: 130888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3732.011279] LNet: Removed LNI 192.168.203.156@tcp [ 3732.818145] Key type .llcrypt unregistered [ 3732.823429] Key type ._llcrypt unregistered [ 3763.544707] Key type ._llcrypt registered [ 3763.548940] Key type .llcrypt registered [ 3763.711197] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 3779.844236] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 3781.224675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3781.318473] alg: No test for adler32 (adler32-zlib) [ 3782.327350] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 3782.556688] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 3784.296391] Key type lgssc registered [ 3785.510322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3826.295444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3826.339599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.660262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3827.689933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3827.771874] Lustre: lustre-MDT0000: new disk, initializing [ 3827.854564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.880063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3831.781137] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3845.182534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.470070] Lustre: 134287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3845.639462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3845.644354] Lustre: Skipped 1 previous similar message [ 3845.860314] Lustre: lustre-MDT0001: new disk, initializing [ 3846.079944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3846.193096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3846.215415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3852.725189] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3866.700551] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3875.039480] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3883.736706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.226704] Lustre: lustre-OST0000: new disk, initializing [ 3884.236966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3884.250954] Lustre: 136259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3884.421199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.937674] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3894.398598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3894.418433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3894.560185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3904.677796] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3910.350459] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3910.547526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3916.515834] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3916.811215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3920.370925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.378537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.395191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3925.603961] Lustre: server umount lustre-OST0000 complete [ 3930.601891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.610077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.622120] Lustre: Skipped 2 previous similar messages [ 3930.648328] Lustre: Skipped 2 previous similar messages [ 3935.712588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3935.720220] Lustre: Skipped 1 previous similar message [ 3936.511055] Lustre: server umount lustre-MDT0000 complete [ 3940.833604] LustreError: 134300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.843167] LustreError: 134300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3941.177192] LustreError: 134280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052785 with bad export cookie 17663831883683209355 [ 3941.181405] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3941.186806] LustreError: 134280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3941.511346] Lustre: server umount lustre-MDT0001 complete [ 3952.015503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.408816] LustreError: 138481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.516461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.738339] LustreError: 138482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.286887] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3962.867206] LustreError: 138481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.971597] LustreError: 138482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.718801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.329161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3975.099294] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 3985.758348] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3992.615797] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3999.786561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4000.296863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.268572] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4015.331960] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4020.162699] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4023.721469] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4028.613309] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4028.866065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4038.579371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.842594] Lustre: lustre-OST0001: new disk, initializing [ 4038.847557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4038.853256] Lustre: 141875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4038.959910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4045.579283] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4047.484781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4047.501800] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4047.633647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4054.276267] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4054.478558] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4059.124324] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4059.347365] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4064.302587] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9991c5d15000.ost_server_uuid 50 [ 4065.689914] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9991c5d15000.ost_server_uuid in IDLE state after 0 sec [ 4076.564566] Lustre: server umount lustre-OST0001 complete [ 4078.561612] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.564214] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.575262] LustreError: Skipped 2 previous similar messages [ 4078.576921] LustreError: 140306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.590597] Lustre: Skipped 1 previous similar message [ 4078.617790] LustreError: 140306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4079.583951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4079.595921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.150472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.161225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.176970] Lustre: Skipped 1 previous similar message [ 4083.688538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.697363] Lustre: Skipped 2 previous similar messages [ 4085.831174] Lustre: server umount lustre-OST0000 complete [ 4092.385349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4092.399525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.413589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.701635] Lustre: server umount lustre-MDT0000 complete [ 4099.047570] LustreError: 139210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4099.078134] LustreError: 139210:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4101.533749] LustreError: 138461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052945 with bad export cookie 17663831883683210237 [ 4101.540602] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4101.549558] LustreError: 138461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4102.003232] Lustre: server umount lustre-MDT0001 complete [ 4121.974535] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4125.089195] Key type lgssc unregistered [ 4125.448182] LNet: 144877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4125.457738] LNetError: 144877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4125.484244] LNet: Removed LNI 192.168.203.156@tcp [ 4126.383401] Key type .llcrypt unregistered [ 4126.385900] Key type ._llcrypt unregistered [ 4170.093985] Key type ._llcrypt registered [ 4170.096646] Key type .llcrypt registered [ 4170.210611] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4193.611257] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4194.885040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4195.006536] alg: No test for adler32 (adler32-zlib) [ 4196.121816] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 4196.319573] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4198.031226] Key type lgssc registered [ 4199.967224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4247.840082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4247.913768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.242238] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4249.289305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4249.393913] Lustre: lustre-MDT0000: new disk, initializing [ 4249.557621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4249.578441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4254.145464] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4264.569733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.722402] Lustre: 149036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4264.826747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4264.834759] Lustre: Skipped 1 previous similar message [ 4264.943496] Lustre: lustre-MDT0001: new disk, initializing [ 4265.074437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.140645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4265.159357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4269.301267] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4277.565770] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4283.463429] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4290.571202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.977709] Lustre: lustre-OST0000: new disk, initializing [ 4290.986850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4290.996543] Lustre: 151006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4291.121204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.091403] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4299.840027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4299.854504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4299.961935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4308.048673] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4313.199989] Lustre: Failing over lustre-OST0000 [ 4313.389725] Lustre: server umount lustre-OST0000 complete [ 4331.496465] Lustre: 147256:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053159/real 1789053159] req@00000000f5114c3f x1875958080504576/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789053175 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4331.552272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4331.999559] Lustre: 147255:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053159/real 1789053159] req@000000002617da0c x1875958080504704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789053175 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4333.087945] Lustre: 147257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053159/real 1789053159] req@00000000ef0b9b40 x1875958080505344/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789053175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4333.169519] Lustre: 147257:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4335.218500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4336.001037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4336.032081] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4337.008629] Lustre: 147257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053164/real 1789053164] req@00000000807e4160 x1875958080506368/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789053180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4341.103091] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4342.114089] Lustre: 147257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053169/real 1789053169] req@000000005ce1b483 x1875958080507392/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789053185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4342.147439] Lustre: 147257:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4343.162439] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4343.174911] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4344.230171] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4348.216824] Lustre: Failing over lustre-MDT0000 [ 4348.262995] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 4348.501900] Lustre: server umount lustre-MDT0000 complete [ 4351.464067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.476015] Lustre: Skipped 2 previous similar messages [ 4356.608549] LustreError: 150050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.652934] LustreError: 150050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4358.471703] LustreError: 149043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.702030] LustreError: 149048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.719036] LustreError: 149048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4366.835214] LustreError: 149049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.853890] LustreError: 149049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4367.841583] Lustre: 147256:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053195/real 1789053195] req@000000005ce1b483 x1875958080520192/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789053211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4367.886378] Lustre: 147256:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4367.900947] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.644923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4372.962900] LustreError: 149049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.001369] LustreError: 149049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4378.090703] LustreError: 147254:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d717c0d5 x1875958080526336/t0(0) o250->MGC192.168.203.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4378.404084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.470191] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4378.948823] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4382.898963] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4383.726689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4383.748621] Lustre: Skipped 1 previous similar message [ 4383.783811] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4383.841483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4393.954593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.964738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4393.976615] Lustre: Skipped 2 previous similar messages [ 4393.994541] Lustre: Skipped 1 previous similar message [ 4399.085143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4399.094358] Lustre: Skipped 1 previous similar message [ 4404.205070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4404.210734] Lustre: Skipped 1 previous similar message [ 4406.239155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4406.423954] Lustre: server umount lustre-OST0000 complete [ 4414.435632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4414.439842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4414.465630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4414.493283] Lustre: Skipped 1 previous similar message [ 4414.523339] Lustre: Skipped 1 previous similar message [ 4416.839192] Lustre: server umount lustre-MDT0000 complete [ 4419.564430] LustreError: 153622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.595293] LustreError: 153622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4420.729205] LustreError: 151004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053264 with bad export cookie 4129461132135849648 [ 4420.734166] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.746128] LustreError: 151004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4421.107064] Lustre: server umount lustre-MDT0001 complete [ 4434.451401] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 4437.382828] Key type lgssc unregistered [ 4437.668914] LNet: 154664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4437.676306] LNetError: 154664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4437.691225] LNet: Removed LNI 192.168.203.156@tcp [ 4438.650821] Key type .llcrypt unregistered [ 4438.654173] Key type ._llcrypt unregistered [ 4474.445934] Key type ._llcrypt registered [ 4474.448638] Key type .llcrypt registered [ 4474.584230] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4475.567379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4475.594114] alg: No test for adler32 (adler32-zlib) [ 4476.586317] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 4476.837805] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 4478.559287] Key type lgssc registered [ 4479.935245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4488.171616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4488.203811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.730933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.005773] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4499.944785] LustreError: 156183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.980739] LustreError: 156183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4503.777224] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4505.062574] LustreError: 156182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.839123] LustreError: 156211:0:(osp_object.c:623:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4507.849324] LustreError: 156211:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4507.857046] LustreError: 156211:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4508.199248] Lustre: server umount lustre-MDT0000 complete [ 4523.301476] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4533.215319] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4567.901044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.202598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4568.227731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4568.311631] Lustre: lustre-MDT0000: new disk, initializing [ 4568.386853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.405427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4572.871349] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4584.764012] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.963628] Lustre: 159515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4585.058612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4585.064618] Lustre: Skipped 1 previous similar message [ 4585.162407] Lustre: lustre-MDT0001: new disk, initializing [ 4585.322570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4585.394172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4585.421434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4591.047119] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4601.324646] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4607.223137] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4614.467581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.771466] Lustre: lustre-OST0000: new disk, initializing [ 4614.774517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4614.780914] Lustre: 161481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4614.855790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4621.302135] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 4623.929778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4623.949422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4624.102556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4631.353235] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4636.219438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4639.366720] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:18:00 (1789053480) [ 4895.716861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.717543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.741512] Lustre: Skipped 1 previous similar message [ 4895.759824] Lustre: Skipped 2 previous similar messages [ 4899.807268] Lustre: server umount lustre-MDT0000 complete [ 4900.834116] LustreError: 159520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4900.873946] LustreError: 159520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4905.237431] LustreError: 161480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053749 with bad export cookie 1766368238352096778 [ 4905.241141] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.256787] LustreError: 161480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4905.960098] LustreError: 163277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.964424] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.039044] Lustre: Skipped 1 previous similar message [ 4906.055550] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4911.075505] LustreError: 159521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.076931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4911.864813] Lustre: server umount lustre-MDT0001 complete [ 4922.596058] Lustre: server umount lustre-OST0000 complete [ 4935.964207] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 4954.576783] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 4999.135874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.473334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4999.543434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4999.718410] Lustre: lustre-MDT0000: new disk, initializing [ 4999.952523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.017060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5005.278342] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5016.956286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.043709] Lustre: 166170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5017.120934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5017.126610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5017.128873] Lustre: Skipped 1 previous similar message [ 5017.221802] Lustre: lustre-MDT0001: new disk, initializing [ 5017.374344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.461953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5017.491401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5021.980115] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5032.977051] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5039.193397] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5045.839792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5046.214248] Lustre: lustre-OST0000: new disk, initializing [ 5046.218965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5046.235400] Lustre: 168141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5046.353138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.023084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5054.039993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5054.179416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5054.769800] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5067.759139] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5073.129497] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5073.454989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5078.409912] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5078.587397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5085.155104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.156300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.174657] LustreError: Skipped 1 previous similar message [ 5085.178050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.180017] Lustre: Skipped 1 previous similar message [ 5094.879526] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5095.058713] Lustre: server umount lustre-OST0000 complete [ 5099.489162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5099.500794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.523551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5099.535362] Lustre: Skipped 3 previous similar messages [ 5113.311352] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5113.490339] Lustre: server umount lustre-MDT0000 complete [ 5115.873202] LustreError: 167733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.901499] LustreError: 167733:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5117.311035] LustreError: 168151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053961 with bad export cookie 1766368238352255615 [ 5117.313918] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.318885] LustreError: 168151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5118.129719] Lustre: server umount lustre-MDT0001 complete [ 5128.993099] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:26:10 (1789053970) [ 5149.924304] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5162.015653] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5204.521521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.711660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5204.753288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5204.926919] Lustre: lustre-MDT0000: new disk, initializing [ 5205.015142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.046742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5210.932355] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5221.667776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.824301] Lustre: 172905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5222.050563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5226.760191] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5236.777887] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5243.596889] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5251.806065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.057934] Lustre: lustre-OST0000: new disk, initializing [ 5252.061705] Lustre: Skipped 1 previous similar message [ 5252.065381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5252.076203] Lustre: Skipped 2 previous similar messages [ 5252.082289] Lustre: 174881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5258.392107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5258.411814] Lustre: Skipped 1 previous similar message [ 5258.417162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5258.605542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5260.029245] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5272.761839] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5284.322369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.322520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.329761] LustreError: Skipped 1 previous similar message [ 5284.330290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5284.330294] Lustre: Skipped 6 previous similar messages [ 5284.363181] Lustre: Skipped 2 previous similar messages [ 5286.132384] Lustre: server umount lustre-OST0000 complete [ 5294.048783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5294.056491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.287125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5304.452088] Lustre: server umount lustre-MDT0000 complete [ 5304.800978] LustreError: 174583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.826555] LustreError: 174583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5308.092087] LustreError: 172897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054152 with bad export cookie 1766368238352256497 [ 5308.101973] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.108418] LustreError: 172897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5308.364525] Lustre: server umount lustre-MDT0001 complete [ 5315.885860] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:29:17 (1789054157) [ 5333.295907] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 5344.054754] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5377.071331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.285739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5377.305988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5377.396211] Lustre: lustre-MDT0000: new disk, initializing [ 5377.450923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.455836] Lustre: Skipped 2 previous similar messages [ 5377.467313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5381.601745] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5391.475328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.641926] Lustre: 179207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5391.995510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5396.698505] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5405.692545] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5411.045507] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.026028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.432387] Lustre: 181184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5425.250600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5425.393764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5425.469417] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5434.290663] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5440.997147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.004976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5441.009051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5441.009060] Lustre: Skipped 5 previous similar messages [ 5441.026787] Lustre: Skipped 2 previous similar messages [ 5441.058454] LustreError: Skipped 1 previous similar message [ 5455.329451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.564033] Lustre: server umount lustre-OST0000 complete [ 5465.832195] Lustre: server umount lustre-MDT0000 complete [ 5466.593286] LustreError: 179220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.612309] LustreError: 179220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5469.378423] LustreError: 180009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054313 with bad export cookie 1766368238352257575 [ 5469.385169] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.390626] LustreError: 180009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5469.895426] Lustre: server umount lustre-MDT0001 complete [ 5480.744316] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5483.680311] Key type lgssc unregistered [ 5484.071324] LNet: 183041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5484.085170] LNetError: 183041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5484.103466] LNet: Removed LNI 192.168.203.156@tcp [ 5485.141373] Key type .llcrypt unregistered [ 5485.144964] Key type ._llcrypt unregistered [ 5517.193764] Key type ._llcrypt registered [ 5517.195824] Key type .llcrypt registered [ 5517.280402] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5518.597086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5518.700437] alg: No test for adler32 (adler32-zlib) [ 5519.830942] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5520.038625] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5521.687210] Key type lgssc registered [ 5522.743368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5531.402662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5531.444945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.108484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.855716] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5542.370046] LustreError: 184454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.402239] LustreError: 184454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5547.490168] LustreError: 184455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.604166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.988953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5553.545284] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5560.060989] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5564.939981] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5571.039422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5571.376735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.242882] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5577.530296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5583.459227] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5603.317439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.317921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.318898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.360716] Lustre: Skipped 1 previous similar message [ 5603.403960] Lustre: Skipped 1 previous similar message [ 5604.793320] Lustre: server umount lustre-OST0000 complete [ 5610.464434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5610.472487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.481798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.538432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.543044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.683985] Lustre: server umount lustre-MDT0000 complete [ 5618.656536] LustreError: 184454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.681686] LustreError: 184454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5621.296108] LustreError: 184434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054465 with bad export cookie 18288313223246941475 [ 5621.302388] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5621.309602] LustreError: 184434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5621.620945] Lustre: server umount lustre-MDT0001 complete [ 5632.590512] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 5636.533123] Key type lgssc unregistered [ 5636.895435] LNet: 188010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5636.911282] LNetError: 188010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5636.942505] LNet: Removed LNI 192.168.203.156@tcp [ 5637.721942] Key type .llcrypt unregistered [ 5637.725565] Key type ._llcrypt unregistered [ 5665.174388] Key type ._llcrypt registered [ 5665.176612] Key type .llcrypt registered [ 5665.383419] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 5667.235161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5667.436214] alg: No test for adler32 (adler32-zlib) [ 5668.722984] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 5669.282379] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 5671.111239] Key type lgssc registered [ 5673.941356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5684.994688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5685.010437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.847507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.977485] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5697.042258] LustreError: 189426:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.081728] LustreError: 189426:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5702.113709] LustreError: 189427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5704.609913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.207532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5710.332868] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5718.056149] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5723.475293] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5730.356126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.611256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.649164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5736.767157] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5743.522707] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5750.619681] Lustre: lustre-OST0000: Not available for connect from 192.168.203.56@tcp (stopping) [ 5752.290017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.293438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.310384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.325615] Lustre: Skipped 1 previous similar message [ 5752.334537] Lustre: Skipped 1 previous similar message [ 5755.594599] Lustre: server umount lustre-OST0000 complete [ 5763.409887] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9991c480b000.ost_server_uuid 50 [ 5765.196661] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5781.984443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5782.001283] LustreError: Skipped 1 previous similar message [ 5782.008559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.048030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5783.010890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.044645] Lustre: server umount lustre-MDT0000 complete [ 5788.128978] LustreError: 189421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.162128] LustreError: 189421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5788.699462] LustreError: 189407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054632 with bad export cookie 16223445937257336491 [ 5788.703451] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5788.715499] LustreError: 189407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5789.100641] Lustre: server umount lustre-MDT0001 complete [ 5797.861482] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:37:19 (1789054639) [ 5804.662557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.013678] LustreError: 193339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.068061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.565830] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5810.144976] LustreError: 193340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.980254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.462204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5822.901491] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5830.206877] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5836.212175] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5842.067042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5842.375186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5847.928353] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5849.594444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5855.327930] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5862.409785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5862.648935] Lustre: lustre-OST0001: new disk, initializing [ 5862.652921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5862.658955] Lustre: 196180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5862.737278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5868.608227] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5870.687545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5870.696120] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5870.811492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5877.493859] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5880.289508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5880.295099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5880.316434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5880.328337] Lustre: Skipped 2 previous similar messages [ 5886.228814] Lustre: server umount lustre-OST0000 complete [ 5886.446243] LustreError: 195164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5886.473130] LustreError: 195164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5893.125662] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5893.389525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5898.469342] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5898.739248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5903.670527] LustreError: 195163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5903.698890] LustreError: 195163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5917.157196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5917.158858] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5917.168759] LustreError: Skipped 1 previous similar message [ 5917.170775] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5917.188406] Lustre: Skipped 2 previous similar messages [ 5917.219542] Lustre: Skipped 4 previous similar messages [ 5920.025791] Lustre: server umount lustre-OST0001 complete [ 5925.856592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5925.866481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.879172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5930.518340] Lustre: server umount lustre-MDT0000 complete [ 5933.937032] LustreError: 193322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054777 with bad export cookie 16223445937257337562 [ 5933.944156] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.960171] LustreError: 193322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5934.363784] Lustre: server umount lustre-MDT0001 complete [ 5957.101420] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:39:59 (1789054799) [ 5963.447419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.882459] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5963.904881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5964.100740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5968.555973] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5976.907675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5976.993091] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5977.022302] Lustre: 199619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5982.078949] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 5989.594366] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5995.940788] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6002.500596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6002.721918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6009.012054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6009.317577] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6016.543277] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6023.168091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.246647] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6031.168164] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6032.515222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6038.269197] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6043.110943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6043.114383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6043.134249] Lustre: Skipped 2 previous similar messages [ 6043.146442] Lustre: Skipped 2 previous similar messages [ 6054.879144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6055.008685] Lustre: server umount lustre-OST0000 complete [ 6058.472212] LustreError: 201471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.493219] LustreError: 201471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6073.314345] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6076.365915] Lustre: server umount lustre-OST0001 complete [ 6084.065735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.067923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.107548] Lustre: Skipped 3 previous similar messages [ 6084.119897] Lustre: Skipped 9 previous similar messages [ 6086.604747] Lustre: server umount lustre-MDT0000 complete [ 6090.713083] LustreError: 202504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054934 with bad export cookie 16223445937257338969 [ 6090.722123] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.732529] LustreError: 202504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6091.432605] Lustre: server umount lustre-MDT0001 complete [ 6116.493628] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:42:38 (1789054958) [ 6133.613747] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 6146.297270] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 6191.087572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.318566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6191.336499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6191.408579] Lustre: lustre-MDT0000: new disk, initializing [ 6191.562773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6191.571138] Lustre: Skipped 3 previous similar messages [ 6191.605820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6196.670584] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6209.340173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.514966] Lustre: 207715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6209.601621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6209.606563] Lustre: Skipped 1 previous similar message [ 6209.689583] Lustre: lustre-MDT0001: new disk, initializing [ 6209.857489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6209.868646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6215.081430] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6227.524186] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6234.329395] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6241.365970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.756942] Lustre: lustre-OST0000: new disk, initializing [ 6241.761568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6241.769931] Lustre: 209688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6247.817270] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6249.547847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6249.555664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6249.674844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6256.182600] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6261.357738] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6261.621526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6266.258443] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6266.478578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6270.437146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.447394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6270.463672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6270.471250] LustreError: Skipped 2 previous similar messages [ 6270.482225] Lustre: Skipped 1 previous similar message [ 6274.606826] Lustre: server umount lustre-OST0000 complete [ 6284.448262] Lustre: server umount lustre-MDT0000 complete [ 6285.793633] LustreError: 207729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6285.820322] LustreError: 207729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6288.346929] LustreError: 207709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055132 with bad export cookie 16223445937257340754 [ 6288.351119] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.359384] LustreError: 207709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6288.825779] Lustre: server umount lustre-MDT0001 complete [ 6297.194921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.481213] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6310.744704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6315.341552] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6322.340127] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6327.990880] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6334.637693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6334.970307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.974905] Lustre: Skipped 4 previous similar messages [ 6341.258265] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6348.059880] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6352.566797] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6352.802961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6357.426608] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6357.591382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6362.598622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6362.604540] LustreError: Skipped 1 previous similar message [ 6373.343950] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6373.629495] Lustre: server umount lustre-OST0000 complete [ 6379.583718] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6384.042535] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6413.419992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6420.527771] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6427.114142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.132242] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.150771] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6427.167320] Lustre: Skipped 1 previous similar message [ 6428.390682] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6432.232180] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6437.349806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6437.357200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.378353] Lustre: Skipped 5 previous similar messages [ 6437.386037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6437.389347] Lustre: Skipped 9 previous similar messages [ 6442.739779] Lustre: server umount lustre-OST0000 complete [ 6452.558650] Lustre: server umount lustre-MDT0000 complete [ 6452.705562] LustreError: 211904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.730103] LustreError: 211904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6456.213729] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055300 with bad export cookie 16223445937257341643 [ 6456.214514] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6456.223019] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6456.636355] Lustre: server umount lustre-MDT0001 complete [ 6465.107735] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:48:27 (1789055307) [ 6471.402573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.723663] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6485.184229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.048476] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6497.453779] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6502.811684] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6508.932383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6515.133742] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6523.089734] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6527.560518] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6527.725795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6532.241810] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6532.447575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6538.854810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6539.120785] Lustre: lustre-OST0001: new disk, initializing [ 6539.127166] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6539.137497] Lustre: 221253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6546.051458] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6547.083100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6547.093042] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6547.179345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6555.472736] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6560.536922] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6560.900619] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6567.049146] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6567.344231] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6573.030684] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6573.048379] LustreError: Skipped 2 previous similar messages [ 6575.464197] Lustre: server umount lustre-OST0001 complete [ 6581.672881] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6581.953563] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6587.697960] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6587.975274] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6618.169695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6618.398650] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6618.409557] Lustre: Skipped 5 previous similar messages [ 6623.719157] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6623.742857] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6623.745157] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6623.766857] Lustre: Skipped 1 previous similar message [ 6627.674565] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6636.692956] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6638.441995] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6646.895649] Lustre: server umount lustre-OST0001 complete [ 6670.559712] Lustre: server umount lustre-OST0000 complete [ 6689.247149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6689.520616] Lustre: server umount lustre-MDT0000 complete [ 6694.029165] LustreError: 221252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055537 with bad export cookie 16223445937257342609 [ 6694.030656] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.052420] LustreError: 221252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6701.463516] Lustre: server umount lustre-MDT0001 complete [ 6730.844352] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:52:52 (1789055572) [ 6738.742690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.987755] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6739.034817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6745.476685] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6756.725045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.076715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6757.174630] Lustre: 226500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6765.331990] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6775.140554] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6784.020278] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6792.610844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6792.946667] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6801.070649] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6813.498477] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.846718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.971489] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6834.880483] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6844.038653] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6846.256359] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6852.204294] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6852.444585] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6857.922822] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6858.220095] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6864.377233] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9991c5d17000.ost_server_uuid 50 [ 6866.188439] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9991c5d17000.ost_server_uuid in IDLE state after 0 sec [ 6867.301499] Lustre: Permanently deactivating lustre-OST0001 [ 6867.323655] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6871.135970] Lustre: Permanently reactivating lustre-OST0001 [ 6879.718424] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6879.719159] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.738966] LustreError: Skipped 3 previous similar messages [ 6879.744165] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6879.762088] Lustre: Skipped 12 previous similar messages [ 6879.805769] Lustre: Skipped 20 previous similar messages [ 6881.892427] Lustre: server umount lustre-OST0001 complete [ 6884.872496] LustreError: 228345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.896859] LustreError: 228345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 6892.511812] Lustre: server umount lustre-OST0000 complete [ 6904.016263] Lustre: server umount lustre-MDT0000 complete [ 6908.627620] LustreError: 226492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055752 with bad export cookie 16223445937257343897 [ 6908.630456] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.646127] LustreError: 226492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6909.145606] Lustre: server umount lustre-MDT0001 complete [ 6938.331975] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:56:19 (1789055779) [ 6946.776900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.154400] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6947.221881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6947.243042] Lustre: Skipped 5 previous similar messages [ 6953.212833] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6963.140332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.298924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6963.375269] Lustre: 233199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6969.308450] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6978.260790] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6984.082965] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6990.328143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6990.491424] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6990.505842] Lustre: Permanently deactivating lustre-OST0000 [ 6990.514825] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6996.474779] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 6997.554834] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6997.678726] LustreError: 188734:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bfef1314 x1875959625495808/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7004.229388] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7011.220579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.308506] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7019.085315] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7020.721930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7028.623888] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7030.571932] Lustre: Permanently reactivating lustre-OST0000 [ 7030.607164] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7030.612948] Lustre: Skipped 2 previous similar messages [ 7030.867969] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7030.882788] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7030.900726] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7030.978782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7046.779515] Lustre: server umount lustre-OST0001 complete [ 7057.024417] Lustre: server umount lustre-OST0000 complete [ 7067.730779] Lustre: server umount lustre-MDT0000 complete [ 7072.010123] LustreError: 233189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055915 with bad export cookie 16223445937257345570 [ 7072.015703] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.029333] LustreError: 233189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7072.785509] Lustre: server umount lustre-MDT0001 complete [ 7085.773806] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7089.278746] Key type lgssc unregistered [ 7089.632886] LNet: 238080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7089.650500] LNetError: 238080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.698144] LNet: Removed LNI 192.168.203.156@tcp [ 7090.932620] Key type .llcrypt unregistered [ 7090.939445] Key type ._llcrypt unregistered [ 7115.497818] Key type ._llcrypt registered [ 7115.510536] Key type .llcrypt registered [ 7115.590238] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7117.206540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7117.270367] alg: No test for adler32 (adler32-zlib) [ 7118.453811] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7118.725622] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 7120.384032] Key type lgssc registered [ 7122.024938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7135.047898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7135.071695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7136.595643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.651498] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7147.002555] LustreError: 239545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7147.047078] LustreError: 239545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7152.097429] LustreError: 239544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7152.170552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.302175] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7152.320593] Lustre: 239531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7152.332771] Lustre: Permanently deactivating lustre-MDT0001 [ 7152.356088] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7152.818142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7157.538743] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7162.419033] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7166.412276] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7172.174980] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7179.518157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7179.875806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.958529] LustreError: 241385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.975253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7187.432905] LustreError: 241387:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.530900] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7192.558498] LustreError: 241385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7195.690308] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7197.681671] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7202.787417] LustreError: 241385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.811314] LustreError: 241385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7202.832531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.241509] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7208.442036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7208.450326] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7209.889970] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7218.818225] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7223.017714] Lustre: Permanently reactivating lustre-MDT0001 [ 7223.037652] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7223.042678] Lustre: Skipped 2 previous similar messages [ 7226.406875] Lustre: lustre-MDT0001: Client 8db528d7-6ddb-442f-935c-0f220bb54d62 (at 192.168.203.56@tcp) reconnecting [ 7230.603692] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.628395] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7230.638346] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7230.675668] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7234.158799] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.252764] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7242.555400] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7242.857677] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7248.398900] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7248.628893] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7251.110588] Lustre: Permanently deactivating lustre-MDT0001 [ 7251.145101] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7251.151039] Lustre: Skipped 2 previous similar messages [ 7259.778059] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7270.369353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.387396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.424357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.976666] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.977909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.991793] Lustre: Skipped 1 previous similar message [ 7275.007491] Lustre: Skipped 3 previous similar messages [ 7276.823996] Lustre: server umount lustre-MDT0000 complete [ 7280.103418] LustreError: 239545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7280.137216] LustreError: 239545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7281.195111] LustreError: 239524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056125 with bad export cookie 17272577454782777043 [ 7281.197223] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7281.208873] LustreError: 239524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7281.557826] Lustre: server umount lustre-MDT0001 complete [ 7287.689380] Lustre: server umount lustre-OST0000 complete [ 7301.247881] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:02:22 (1789056142) [ 7314.861857] Lustre: server umount lustre-OST0001 complete [ 7323.573548] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 7335.682193] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7383.469037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7383.968243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7383.986827] Lustre: Skipped 2 previous similar messages [ 7384.043312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7384.139477] Lustre: lustre-MDT0000: new disk, initializing [ 7384.260660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.294281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7390.819667] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7402.531562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7402.724688] Lustre: 247869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7402.804964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7402.816206] Lustre: Skipped 1 previous similar message [ 7402.921552] Lustre: lustre-MDT0001: new disk, initializing [ 7403.065967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7403.152187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7403.163692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7407.787571] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7418.268795] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7425.948146] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.128577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7434.596753] Lustre: lustre-OST0000: new disk, initializing [ 7434.611202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7434.630059] Lustre: 249837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7434.760962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.499968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7441.515207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7441.639530] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7441.681234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7453.008509] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7455.032814] LustreError: 247875:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7464.444067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7464.626235] Lustre: lustre-OST0001: new disk, initializing [ 7464.632068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7464.644656] Lustre: 250951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7465.048655] LustreError: 247875:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7465.083163] LustreError: 247876:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7471.210373] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7471.743523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7471.757145] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7471.924534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7475.175077] LustreError: 247876:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7481.129950] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7482.913374] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7487.464481] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7487.475381] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.486404] LustreError: Skipped 1 previous similar message [ 7487.487981] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7487.530721] Lustre: Skipped 2 previous similar messages [ 7491.362251] Lustre: server umount lustre-OST0001 complete [ 7492.600339] LustreError: 249822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.618085] LustreError: 249822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7513.064055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.078037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7513.080059] Lustre: Skipped 1 previous similar message [ 7513.098744] Lustre: Skipped 2 previous similar messages [ 7516.664337] Lustre: server umount lustre-OST0000 complete [ 7523.301909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.306428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.339718] Lustre: Skipped 1 previous similar message [ 7523.357505] Lustre: Skipped 1 previous similar message [ 7527.212701] Lustre: server umount lustre-MDT0000 complete [ 7531.253913] LustreError: 247860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056375 with bad export cookie 17272577454782779864 [ 7531.264224] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.282204] LustreError: 247860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7537.889532] Lustre: server umount lustre-MDT0001 complete [ 7551.968363] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7554.962459] Key type lgssc unregistered [ 7555.287990] LNet: 252987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7555.294191] LNetError: 252987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7555.313685] LNet: Removed LNI 192.168.203.156@tcp [ 7556.457335] Key type .llcrypt unregistered [ 7556.462352] Key type ._llcrypt unregistered [ 7602.391943] Key type ._llcrypt registered [ 7602.396152] Key type .llcrypt registered [ 7602.530289] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7603.530317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7603.724485] alg: No test for adler32 (adler32-zlib) [ 7604.920679] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7605.311291] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 7607.015219] Key type lgssc registered [ 7608.232189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7616.403718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7616.427344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.697882] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7617.716588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7617.934401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.669783] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7632.396072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.553780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7632.584398] Lustre: 255191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7632.904499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.693543] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7646.821836] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7653.984722] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7663.279962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7663.631940] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7664.110690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7670.606128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7673.492950] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7684.919302] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.432804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.433957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.436929] LustreError: Skipped 1 previous similar message [ 7742.437911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.464706] Lustre: Skipped 1 previous similar message [ 7745.896775] Lustre: server umount lustre-OST0000 complete [ 7750.638963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7763.611792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7763.961519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.843362] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7777.693445] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7782.319567] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7782.333077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7782.334249] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7782.348431] Lustre: Skipped 1 previous similar message [ 7818.215121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7818.220787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.224039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7818.224047] Lustre: Skipped 1 previous similar message [ 7818.225455] LustreError: Skipped 1 previous similar message [ 7818.273043] Lustre: Skipped 1 previous similar message [ 7823.351419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7823.362489] Lustre: Skipped 2 previous similar messages [ 7824.029889] Lustre: server umount lustre-OST0000 complete [ 7828.452918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7828.464812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7828.468728] Lustre: Skipped 1 previous similar message [ 7828.487158] Lustre: Skipped 1 previous similar message [ 7833.568771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7833.597416] Lustre: Skipped 1 previous similar message [ 7834.351366] Lustre: server umount lustre-MDT0000 complete [ 7838.921144] LustreError: 255182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056682 with bad export cookie 17411687174619529465 [ 7838.939152] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7838.939256] LustreError: 255182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7839.455860] Lustre: server umount lustre-MDT0001 complete [ 7855.156756] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 7860.446241] Key type lgssc unregistered [ 7860.826666] LNet: 260619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7860.833854] LNetError: 260619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7860.858274] LNet: Removed LNI 192.168.203.156@tcp [ 7862.069859] Key type .llcrypt unregistered [ 7862.074097] Key type ._llcrypt unregistered [ 7889.956676] Key type ._llcrypt registered [ 7889.960917] Key type .llcrypt registered [ 7890.119842] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 7891.466070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7891.482963] alg: No test for adler32 (adler32-zlib) [ 7892.887836] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 7893.315379] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 7895.104749] Key type lgssc registered [ 7896.968213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7907.988511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7908.013939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.687065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.868900] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7920.100750] LustreError: 262034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7920.145794] LustreError: 262034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7925.217948] LustreError: 262033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.181812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.666597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7932.982193] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7942.309042] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7948.446313] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7955.630197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.989538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.503846] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 7969.853863] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.467221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8006.114663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.124720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8006.132260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8006.138373] Lustre: Skipped 1 previous similar message [ 8011.243402] LustreError: 263859:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8011.370760] Lustre: server umount lustre-OST0000 complete [ 8015.841564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8015.857659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.885553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8015.896601] Lustre: Skipped 1 previous similar message [ 8021.476153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8021.482168] Lustre: Skipped 3 previous similar messages [ 8021.756203] Lustre: server umount lustre-MDT0000 complete [ 8025.911382] LustreError: 262013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056869 with bad export cookie 14373661293246478268 [ 8025.924560] LustreError: 262013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8025.926165] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8026.579302] Lustre: server umount lustre-MDT0001 complete [ 8038.770456] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8042.102856] Key type lgssc unregistered [ 8042.483821] LNet: 266362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8042.512957] LNetError: 266362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8043.568575] LNet: Removed LNI 192.168.203.156@tcp [ 8044.665088] Key type .llcrypt unregistered [ 8044.667552] Key type ._llcrypt unregistered [ 8064.535907] Key type ._llcrypt registered [ 8064.541324] Key type .llcrypt registered [ 8064.744285] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8066.297890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8066.346346] alg: No test for adler32 (adler32-zlib) [ 8067.554841] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 8067.864530] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8069.607187] Key type lgssc registered [ 8071.330364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8080.672348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8080.715097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.449479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8087.463617] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8092.643687] LustreError: 267582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8092.667974] LustreError: 267582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8095.924417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8096.266419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8100.545760] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8108.234445] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8114.265883] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8120.635890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.925142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.956314] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8128.123213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8135.130243] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8147.939531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8147.949079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.966824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8148.967703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8148.969395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8148.991171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8151.884502] Lustre: server umount lustre-OST0000 complete [ 8157.664470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8157.671759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.691299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.701348] Lustre: Skipped 1 previous similar message [ 8161.981579] Lustre: server umount lustre-MDT0000 complete [ 8164.334464] LustreError: 267582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8164.347529] LustreError: 267582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8166.125834] LustreError: 267564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057010 with bad export cookie 14626805408655475266 [ 8166.135920] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8166.138766] LustreError: 267564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8166.479247] Lustre: server umount lustre-MDT0001 complete [ 8179.455413] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8182.479781] Key type lgssc unregistered [ 8182.796407] LNet: 271259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8182.807470] LNetError: 271259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8182.828441] LNet: Removed LNI 192.168.203.156@tcp [ 8183.739970] Key type .llcrypt unregistered [ 8183.743693] Key type ._llcrypt unregistered [ 8206.721066] Key type ._llcrypt registered [ 8206.729565] Key type .llcrypt registered [ 8206.871652] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8207.528678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8207.571967] alg: No test for adler32 (adler32-zlib) [ 8208.687131] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 8208.954871] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8210.720802] Key type lgssc registered [ 8211.717638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8219.492424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8219.515898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.077613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.236348] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8230.375820] LustreError: 272674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8230.416588] LustreError: 272674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8235.488565] LustreError: 272675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8236.367729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8236.947796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8241.609529] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8248.982451] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8255.220976] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8263.649785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.178438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.203866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8272.105398] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8280.669663] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8306.145171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8306.159588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8306.186762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8307.169201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8307.178352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8307.179852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.203039] Lustre: Skipped 1 previous similar message [ 8310.492906] Lustre: server umount lustre-OST0000 complete [ 8317.418656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8317.422164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8317.437542] Lustre: Skipped 1 previous similar message [ 8317.449475] Lustre: Skipped 1 previous similar message [ 8320.602891] Lustre: server umount lustre-MDT0000 complete [ 8322.529974] LustreError: 272675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8322.543858] LustreError: 272675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8324.660770] LustreError: 272655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057168 with bad export cookie 14679080703082031149 [ 8324.675899] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8325.024185] Lustre: server umount lustre-MDT0001 complete [ 8336.395967] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8339.493089] Key type lgssc unregistered [ 8339.805042] LNet: 276979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8339.816411] LNetError: 276979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8339.829306] LNet: Removed LNI 192.168.203.156@tcp [ 8340.590141] Key type .llcrypt unregistered [ 8340.593516] Key type ._llcrypt unregistered [ 8358.357735] Key type ._llcrypt registered [ 8358.360640] Key type .llcrypt registered [ 8358.473491] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8359.941895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8360.029083] alg: No test for adler32 (adler32-zlib) [ 8361.099650] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 8361.476912] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8363.215590] Key type lgssc registered [ 8364.261941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8372.192962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8372.236721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.731446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8378.459812] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8383.972952] LustreError: 278211:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.006162] LustreError: 278211:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8389.091063] LustreError: 278213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.145101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.743121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8395.897623] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8408.360102] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8416.073584] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8423.638147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.038110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8430.711254] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8439.498311] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8443.518608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8459.232526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8459.243987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8459.250743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8459.272869] Lustre: Skipped 1 previous similar message [ 8462.738824] Lustre: server umount lustre-OST0000 complete [ 8467.425978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8467.449984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.460992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.464230] Lustre: Skipped 1 previous similar message [ 8469.472509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8469.472518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8469.495612] Lustre: Skipped 1 previous similar message [ 8473.141680] Lustre: server umount lustre-MDT0000 complete [ 8474.593805] LustreError: 278197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8474.619584] LustreError: 278197:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8477.570724] LustreError: 278178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057321 with bad export cookie 17709475172850655210 [ 8477.574814] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8477.577392] LustreError: 278178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8477.848927] Lustre: server umount lustre-MDT0001 complete [ 8488.719468] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8492.007551] Key type lgssc unregistered [ 8492.495404] LNet: 281914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8492.530544] LNetError: 281914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8492.565900] LNet: Removed LNI 192.168.203.156@tcp [ 8494.085286] Key type .llcrypt unregistered [ 8494.088311] Key type ._llcrypt unregistered [ 8507.926211] Key type ._llcrypt registered [ 8507.930074] Key type .llcrypt registered [ 8508.120614] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8527.171994] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 8546.995445] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8548.299656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8548.552562] alg: No test for adler32 (adler32-zlib) [ 8549.642255] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 8549.899191] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8551.623716] Key type lgssc registered [ 8552.767274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8594.230149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8594.269921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.573471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8595.647509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8595.722832] Lustre: lustre-MDT0000: new disk, initializing [ 8595.852261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8595.870384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8601.680849] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8613.257237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.402543] Lustre: 285471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8613.476218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8613.483091] Lustre: Skipped 1 previous similar message [ 8613.582388] Lustre: lustre-MDT0001: new disk, initializing [ 8613.730177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8613.786104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8613.805910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8618.647992] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8628.255552] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8634.139555] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8640.728049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8641.046082] Lustre: lustre-OST0000: new disk, initializing [ 8641.057903] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8641.070690] Lustre: 287438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8641.192933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8648.287809] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8650.834174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8650.863066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8650.998210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8660.403297] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8665.642263] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8665.871211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8670.744862] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8671.035593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8676.835025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.835964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.866160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.874359] LustreError: Skipped 1 previous similar message [ 8676.892102] Lustre: Skipped 1 previous similar message [ 8679.419890] Lustre: server umount lustre-OST0000 complete [ 8685.539586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8685.550808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.559991] Lustre: Skipped 1 previous similar message [ 8685.579282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8687.072617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.074711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8687.111968] Lustre: Skipped 1 previous similar message [ 8692.196395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8692.207540] Lustre: Skipped 1 previous similar message [ 8696.289525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8696.299351] Lustre: Skipped 1 previous similar message [ 8697.312567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8697.793698] Lustre: server umount lustre-MDT0000 complete [ 8702.494846] LustreError: 285462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057546 with bad export cookie 12839387746428424731 [ 8702.498072] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8702.542000] LustreError: 285462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8703.419890] Lustre: server umount lustre-MDT0001 complete [ 8715.098932] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:25:56 (1789057556) [ 8723.628808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.151391] LustreError: 289854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8724.171104] LustreError: 289854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8724.219590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8728.951798] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8729.573673] LustreError: 289855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8734.690546] LustreError: 289854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.391246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.797295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8743.613137] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8752.807093] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8759.384782] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8767.964909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.248913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8775.010120] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8783.139768] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8794.089767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.094804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8794.115915] Lustre: Skipped 1 previous similar message [ 8798.732626] Lustre: server umount lustre-OST0000 complete [ 8804.336101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.359644] Lustre: Skipped 1 previous similar message [ 8809.290835] Lustre: server umount lustre-MDT0000 complete [ 8809.440832] LustreError: 289851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.449233] LustreError: 289851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8813.862349] LustreError: 289834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057657 with bad export cookie 12839387746428425613 [ 8813.873055] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8813.880061] LustreError: 289834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8814.285773] Lustre: server umount lustre-MDT0001 complete [ 8825.554782] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 8829.359988] Key type lgssc unregistered [ 8829.736452] LNet: 293389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8829.753475] LNetError: 293389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8829.784521] LNet: Removed LNI 192.168.203.156@tcp [ 8830.736873] Key type .llcrypt unregistered [ 8830.738612] Key type ._llcrypt unregistered [ 8860.680888] Key type ._llcrypt registered [ 8860.685421] Key type .llcrypt registered [ 8860.831430] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 8875.873502] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 8877.045269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8877.081909] alg: No test for adler32 (adler32-zlib) [ 8878.181498] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 8878.406651] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 8880.080154] Key type lgssc registered [ 8881.530979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8927.655374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8927.684849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.914219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8928.936649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8928.985097] Lustre: lustre-MDT0000: new disk, initializing [ 8929.071783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8929.091379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8933.803543] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8949.951097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.094491] Lustre: 297215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8950.150990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8950.156090] Lustre: Skipped 1 previous similar message [ 8950.249377] Lustre: lustre-MDT0001: new disk, initializing [ 8950.374769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8950.413729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8950.428941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8956.955749] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 8967.799978] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8973.994634] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8982.397061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.948691] Lustre: lustre-OST2710: new disk, initializing [ 8982.963068] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8982.980784] Lustre: 299185:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8983.124460] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8988.786551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8988.792759] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8988.877283] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8990.237402] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9001.184547] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9009.132104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.404442] Lustre: lustre-OST03e8: new disk, initializing [ 9009.413855] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9009.431938] Lustre: 300246:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9009.569638] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9015.991737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9016.015706] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9016.240091] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9016.609790] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9026.277122] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9033.818914] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9034.087028] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9038.869905] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9039.176463] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9044.895544] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9045.176738] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9050.353310] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9050.565397] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9057.251168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.257888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.264865] Lustre: Skipped 3 previous similar messages [ 9057.283223] Lustre: Skipped 3 previous similar messages [ 9057.495304] Lustre: server umount lustre-MDT0000 complete [ 9061.050209] LustreError: 297208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057904 with bad export cookie 10363416154391558228 [ 9061.060741] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.067536] LustreError: 297208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9061.369187] Lustre: server umount lustre-MDT0001 complete [ 9076.742093] Lustre: server umount lustre-OST2710 complete [ 9078.751703] Lustre: 295015:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789057906/real 1789057906] req@0000000074f44438 x1875962989975680/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789057922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9078.780983] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9081.888623] Lustre: server umount lustre-OST03e8 complete [ 9103.769105] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [ 9114.735970] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9151.901854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.220471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9152.246411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9152.326838] Lustre: lustre-MDT0000: new disk, initializing [ 9152.400366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9152.421840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9157.345640] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9168.163187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.264670] Lustre: 305500:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9168.345636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9168.356667] Lustre: Skipped 1 previous similar message [ 9168.450369] Lustre: lustre-MDT0001: new disk, initializing [ 9168.607196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9168.681105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9168.697734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9172.922405] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9181.828424] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9186.981652] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9193.562677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.891312] Lustre: lustre-OST0000: new disk, initializing [ 9193.898439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9193.905790] Lustre: 307466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9193.994314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9199.450372] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9201.734207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9201.746367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9201.862831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9208.047723] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9212.063167] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9212.322337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9217.262342] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9217.480848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9222.624403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9222.624866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.632849] LustreError: Skipped 1 previous similar message [ 9222.634841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.681025] Lustre: Skipped 1 previous similar message [ 9227.753696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9227.763351] Lustre: Skipped 2 previous similar messages [ 9232.872027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9233.375121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9233.559360] Lustre: server umount lustre-OST0000 complete [ 9237.987948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.988941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9238.003442] Lustre: Skipped 1 previous similar message [ 9238.022874] Lustre: Skipped 2 previous similar messages [ 9248.227809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9248.238854] Lustre: Skipped 3 previous similar messages [ 9251.299318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9251.616221] Lustre: server umount lustre-MDT0000 complete [ 9253.346679] LustreError: 305511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.387180] LustreError: 305511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9254.946437] LustreError: 307476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058098 with bad export cookie 10363416154391559901 [ 9254.949699] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9254.957019] LustreError: 307476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9255.382065] Lustre: server umount lustre-MDT0001 complete [ 9264.026720] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:35:05 (1789058105) [ 9265.548249] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9267.380417] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:35:09 (1789058109) [ 9271.109526] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9308.571699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.895598] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9308.936916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9309.194931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9314.129382] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9323.511164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9323.683581] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9323.728215] Lustre: 311325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9328.682287] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9350.121474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9350.280870] Lustre: Denying initial registration attempt from nid 192.168.203.156@tcp, specified as failover [ 9350.290478] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9350.305766] LustreError: 312778:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9350.317817] LustreError: 312778:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9350.323205] LustreError: 312778:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9350.372981] Lustre: server umount lustre-OST0000 complete [ 9350.383491] LustreError: 312778:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9358.817064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.834486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9358.836887] Lustre: Skipped 1 previous similar message [ 9362.377707] Lustre: server umount lustre-MDT0000 complete [ 9363.937238] LustreError: 311333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.956752] LustreError: 311333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9365.839155] LustreError: 311319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058209 with bad export cookie 10363416154391560790 [ 9365.839767] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9365.855235] LustreError: 311319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9366.342270] Lustre: server umount lustre-MDT0001 complete [ 9376.303740] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9379.187982] Key type lgssc unregistered [ 9379.573473] LNet: 313900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9379.586655] LNetError: 313900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9379.628734] LNet: Removed LNI 192.168.203.156@tcp [ 9380.352195] Key type .llcrypt unregistered [ 9380.360325] Key type ._llcrypt unregistered [ 9391.444718] Key type ._llcrypt registered [ 9391.449753] Key type .llcrypt registered [ 9391.550253] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9392.568992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9392.587430] alg: No test for adler32 (adler32-zlib) [ 9393.696323] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 9394.002573] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9395.751608] Key type lgssc registered [ 9396.787943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9441.116946] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9450.050173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9450.083122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.381930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9451.409603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9451.577372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9455.233222] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9463.251378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.403583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9463.443509] Lustre: 316662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9463.653694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9469.480186] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9479.011207] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9485.980350] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9506.493592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9506.742769] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9506.942251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9512.453898] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9520.012259] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9524.312431] Lustre: server umount lustre-OST0000 complete [ 9530.338513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9530.356294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.371769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.871994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.875075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9536.994443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9537.004918] Lustre: Skipped 2 previous similar messages [ 9538.149532] Lustre: server umount lustre-MDT0000 complete [ 9541.694221] LustreError: 316654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058385 with bad export cookie 17280030392408432770 [ 9541.706220] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9541.715422] LustreError: 316654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9551.486543] Lustre: server umount lustre-MDT0001 complete [ 9562.619981] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9565.522727] Key type lgssc unregistered [ 9565.817427] LNet: 320239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9565.833631] LNetError: 320239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9565.851869] LNet: Removed LNI 192.168.203.156@tcp [ 9566.671090] Key type .llcrypt unregistered [ 9566.673524] Key type ._llcrypt unregistered [ 9591.586473] Key type ._llcrypt registered [ 9591.589408] Key type .llcrypt registered [ 9591.727410] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9593.154642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9593.248058] alg: No test for adler32 (adler32-zlib) [ 9594.300056] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 9594.570984] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9596.240138] Key type lgssc registered [ 9597.374846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9606.781142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9606.811610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.568246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9614.714356] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9618.917228] LustreError: 321653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9618.927712] LustreError: 321653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9624.035080] LustreError: 321654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.454608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.024862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9631.627163] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9638.805882] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9644.008308] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9652.152533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9652.597895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9658.204136] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9664.731582] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9682.406932] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.411948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9682.415033] Lustre: Skipped 2 previous similar messages [ 9682.422160] Lustre: Skipped 2 previous similar messages [ 9687.530118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9687.537335] Lustre: Skipped 1 previous similar message [ 9692.643247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9692.663155] Lustre: Skipped 3 previous similar messages [ 9693.152325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9693.675805] Lustre: server umount lustre-MDT0000 complete [ 9697.765043] LustreError: 322658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9697.787140] LustreError: 322658:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9697.883679] LustreError: 321634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058541 with bad export cookie 678983106439396261 [ 9697.886967] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9697.896057] LustreError: 321634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9698.133708] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [ 9698.140465] Lustre: Skipped 1 previous similar message [ 9702.884761] LustreError: 321649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.894508] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9702.916128] LustreError: 321649:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9702.985364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9704.857741] Lustre: server umount lustre-MDT0001 complete [ 9715.952442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9731.101794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.900193] LustreError: 323487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.913119] LustreError: 323487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9742.967782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.035267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9749.421192] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9752.571878] LustreError: 325274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.592910] LustreError: 325274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9757.678078] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9757.831605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.380265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9762.895757] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9767.947945] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9772.643928] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9774.479451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9780.767870] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9782.577267] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9788.395554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.396660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9788.404443] Lustre: Skipped 1 previous similar message [ 9788.416791] Lustre: Skipped 2 previous similar messages [ 9794.526928] Lustre: server umount lustre-OST0000 complete [ 9799.136570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9799.145993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.747442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9805.245640] Lustre: server umount lustre-MDT0000 complete [ 9808.866150] LustreError: 325278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.892446] LustreError: 325278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9809.157913] LustreError: 321633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058653 with bad export cookie 678983106439403821 [ 9809.169718] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9809.173174] LustreError: 321633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9809.513827] Lustre: server umount lustre-MDT0001 complete [ 9820.710720] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [ 9824.168761] Key type lgssc unregistered [ 9824.497894] LNet: 327947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9824.513988] LNetError: 327947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9824.541184] LNet: Removed LNI 192.168.203.156@tcp [ 9825.444666] Key type .llcrypt unregistered [ 9825.446663] Key type ._llcrypt unregistered [ 9849.456452] Key type ._llcrypt registered [ 9849.458925] Key type .llcrypt registered [ 9849.580306] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [ 9850.871524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9850.924685] alg: No test for adler32 (adler32-zlib) [ 9852.115150] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 9852.486245] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [ 9854.215207] Key type lgssc registered [ 9855.471856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9863.083153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9863.099096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9864.665067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.310279] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9874.919126] LustreError: 329361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.948610] LustreError: 329361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9878.065908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9878.427451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9884.037896] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9892.685535] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9899.438238] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9903.585730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9903.594164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9903.599541] Lustre: Skipped 1 previous similar message [ 9908.711986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.724088] Lustre: Skipped 1 previous similar message [ 9908.977695] Lustre: server umount lustre-MDT0000 complete [ 9912.707662] LustreError: 330977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058756 with bad export cookie 12232286295235307109 [ 9912.714089] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9912.730330] LustreError: 330977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9913.073697] Lustre: server umount lustre-MDT0001 complete [ 9927.368894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.776309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9927.824411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9928.147589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9933.176138] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9942.700262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.867441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9942.890859] Lustre: 331945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9943.312790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9947.998298] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9956.514685] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9963.255552] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9974.720220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9975.009497] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9975.017962] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9975.027687] LustreError: 333832:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9975.049482] LustreError: 333832:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9975.068116] LustreError: 333832:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9975.171666] Lustre: server umount lustre-OST0000 complete [ 9975.176630] LustreError: 333832:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9987.394660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9987.623079] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9987.900477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9994.452252] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [ 9997.957339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10001.732232] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10008.944603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.053562] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10009.111456] Lustre: lustre-OST0001: new disk, initializing [10009.114624] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10009.119616] Lustre: 335374:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10009.194441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10014.964730] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10016.855986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10016.869712] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10016.954929] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10021.371181] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10027.488953] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10027.490810] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.500167] LustreError: Skipped 1 previous similar message [10027.506437] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10027.523532] Lustre: Skipped 2 previous similar messages [10030.715914] Lustre: server umount lustre-OST0001 complete [10032.613408] LustreError: 334299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.652707] LustreError: 334299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10037.728635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.729902] LustreError: 334299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10037.749556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10037.767135] LustreError: 334299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10037.798291] Lustre: Skipped 2 previous similar messages [10040.996766] Lustre: server umount lustre-OST0000 complete [10045.920415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10045.925467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.933629] Lustre: Skipped 1 previous similar message [10045.939327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10051.372566] Lustre: server umount lustre-MDT0000 complete [10053.091724] LustreError: 331954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10053.119116] LustreError: 331954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10054.661946] LustreError: 331940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058898 with bad export cookie 12232286295235307662 [10054.668791] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10054.673403] LustreError: 331940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10054.958913] Lustre: server umount lustre-MDT0001 complete [10064.392826] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10066.942940] Key type lgssc unregistered [10067.203594] LNet: 337333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10067.221723] LNetError: 337333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10067.249737] LNet: Removed LNI 192.168.203.156@tcp [10068.179432] Key type .llcrypt unregistered [10068.181591] Key type ._llcrypt unregistered [10141.573529] Key type ._llcrypt registered [10141.576974] Key type .llcrypt registered [10141.674242] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [10160.716645] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10161.359969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10161.463946] alg: No test for adler32 (adler32-zlib) [10162.580912] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [10162.865281] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10164.513080] Key type lgssc registered [10165.611906] Lustre: Echo OBD driver; http://www.lustre.org/ [10198.481332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10198.535634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.808426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10199.841880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10199.935724] Lustre: lustre-MDT0000: new disk, initializing [10200.039021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10200.061889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10204.919042] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10216.999250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.161164] Lustre: 342265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10217.203537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10217.206941] Lustre: Skipped 1 previous similar message [10217.268854] Lustre: lustre-MDT0001: new disk, initializing [10217.422500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10217.483643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10217.495715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10222.975409] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10232.719698] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10238.214068] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10245.605887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10245.906223] Lustre: lustre-OST0000: new disk, initializing [10245.912410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10245.918050] Lustre: 344232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10246.017058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10252.421653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10252.444089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10252.584864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10253.446897] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10264.113147] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10268.988502] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10269.157898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10273.344345] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10273.530034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10278.369117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10278.386044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.399182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10278.420964] Lustre: Skipped 1 previous similar message [10281.730460] Lustre: server umount lustre-OST0000 complete [10288.623139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.627583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10288.654687] Lustre: Skipped 2 previous similar messages [10288.660971] Lustre: Skipped 1 previous similar message [10291.908644] Lustre: server umount lustre-MDT0000 complete [10295.740467] LustreError: 342255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059139 with bad export cookie 14799439831415673152 [10295.754119] LustreError: 342255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10295.758922] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10296.136343] Lustre: server umount lustre-MDT0001 complete [10303.989408] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:52:26 (1789059146) [10311.103172] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:52:33 (1789059153) [10319.445899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.978736] LustreError: 346987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.990458] LustreError: 346987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10320.066703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.648480] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10325.480093] LustreError: 346988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10325.500767] LustreError: 346988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10329.569653] LustreError: 346987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.651829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.108829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10338.576978] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10346.581846] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10352.275906] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10359.294766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10359.693952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10366.275781] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10373.537048] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10384.071095] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10385.577685] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10387.125099] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10388.827885] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10390.616975] Lustre: Failing over lustre-MDT0000 [10390.877636] LustreError: 346983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10391.006048] Lustre: server umount lustre-MDT0000 complete [10392.549807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.563266] Lustre: Skipped 2 previous similar messages [10395.971076] LustreError: 346984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.992926] LustreError: 346984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10406.199844] LustreError: 346984:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.218508] LustreError: 346984:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10408.928396] Lustre: 340485:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789059236/real 1789059236] req@000000001c0fd588 x1875964336348288/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789059252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10408.959189] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10409.021033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.465575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10418.523583] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10421.561805] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10423.144882] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10423.789484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10423.808077] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10423.850354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10432.610672] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10434.873296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10438.765203] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10441.239127] Lustre: Failing over lustre-MDT0000 [10441.968795] Lustre: server umount lustre-MDT0000 complete [10442.099102] LustreError: 346982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10442.126751] LustreError: 346982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [10444.258642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10444.260124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10444.279614] Lustre: Skipped 1 previous similar message [10444.309842] LustreError: Skipped 1 previous similar message [10460.641789] Lustre: 340486:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789059288/real 1789059288] req@00000000f67cc4e7 x1875964336369024/t0(0) o400->MGC192.168.203.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789059304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10460.653535] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10466.273785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10470.909931] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcd622c1daa4c5bdc [10470.945571] Lustre: MGC192.168.203.156@tcp: Connection restored to 0@lo (at 0@lo) [10470.961853] Lustre: Skipped 2 previous similar messages [10471.376656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10471.441946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10473.784995] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10476.523400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10476.539123] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10476.579498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10476.855977] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10485.903139] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10487.259023] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10496.996572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.999306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10497.023211] Lustre: Skipped 3 previous similar messages [10497.037543] Lustre: Skipped 1 previous similar message [10498.968236] Lustre: server umount lustre-MDT0000 complete [10502.118224] LustreError: 346987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10502.147179] LustreError: 346987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 29 previous similar messages [10503.074475] LustreError: 346969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059346 with bad export cookie 14799439831415675868 [10503.080810] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10503.083657] LustreError: 346969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10503.760035] Lustre: server umount lustre-MDT0001 complete [10518.219544] Lustre: server umount lustre-OST0000 complete [10535.816258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.409593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10541.289875] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10548.581199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.878623] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10560.301548] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10565.993261] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10572.369025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10579.177248] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10581.934126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10586.765474] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10592.229277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10592.242119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10592.244751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10592.249714] Lustre: Skipped 1 previous similar message [10597.357829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10597.368761] Lustre: Skipped 1 previous similar message [10598.326656] Lustre: server umount lustre-OST0000 complete [10607.587953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10607.593927] Lustre: Skipped 4 previous similar messages [10608.652545] Lustre: server umount lustre-MDT0000 complete [10612.705473] LustreError: 355170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.738032] LustreError: 355170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10613.444513] LustreError: 353313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059457 with bad export cookie 14799439831415676610 [10613.455392] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10613.472362] LustreError: 353313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10614.096313] Lustre: server umount lustre-MDT0001 complete [10627.667972] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10630.969909] Key type lgssc unregistered [10631.557865] LNet: 356861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10631.576189] LNetError: 356861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10631.635475] LNet: Removed LNI 192.168.203.156@tcp [10632.899409] Key type .llcrypt unregistered [10632.901912] Key type ._llcrypt unregistered [10666.084238] Key type ._llcrypt registered [10666.086303] Key type .llcrypt registered [10666.182710] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10667.252096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10667.325606] alg: No test for adler32 (adler32-zlib) [10668.483342] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [10668.906407] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10670.655422] Key type lgssc registered [10672.038497] Lustre: Echo OBD driver; http://www.lustre.org/ [10679.795706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10679.811862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.165072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10685.901617] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10691.555924] LustreError: 358377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.579919] LustreError: 358377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10694.472023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.904394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10698.665996] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10705.893703] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10711.472355] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10717.977923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10718.422232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10724.057869] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10726.661677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10731.404310] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10742.248660] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.251057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10742.270580] Lustre: Skipped 1 previous similar message [10743.784762] Lustre: server umount lustre-MDT0000 complete [10747.138042] LustreError: 360869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059591 with bad export cookie 7428988576046903135 [10747.144165] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10747.151354] LustreError: 360869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10747.362213] LustreError: 358372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10747.371391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.381738] Lustre: Skipped 1 previous similar message [10747.666855] Lustre: server umount lustre-MDT0001 complete [10753.821602] Lustre: server umount lustre-OST0000 complete [10768.556880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.038393] LustreError: 362086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.060238] LustreError: 362086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10769.124550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.682206] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10774.497403] LustreError: 362087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10778.594981] LustreError: 362086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.867656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.194941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10787.388748] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10794.756860] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10800.088658] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10806.050728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10806.378593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10812.034855] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10813.593148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10820.195094] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10834.400455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10834.408469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.409333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10834.409338] Lustre: Skipped 2 previous similar messages [10834.480812] Lustre: Skipped 2 previous similar messages [10838.765251] Lustre: server umount lustre-MDT0000 complete [10839.528323] LustreError: 362087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10839.552525] LustreError: 362087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10842.130229] LustreError: 362068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059686 with bad export cookie 7428988576046904248 [10842.132181] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10842.143797] LustreError: 362068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10842.411279] Lustre: server umount lustre-MDT0001 complete [10855.591863] Lustre: server umount lustre-OST0000 complete [10870.973246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10871.553466] LustreError: 366009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.684828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10876.541748] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10884.592848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.680887] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10895.729052] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10901.613240] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10908.644447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10909.113833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10909.122789] Lustre: Skipped 1 previous similar message [10914.931889] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [10916.270639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10922.403968] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10930.661161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10930.667077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.685600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10930.692358] Lustre: Skipped 2 previous similar messages [10931.685254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10933.259874] Lustre: server umount lustre-OST0000 complete [10941.410131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10941.416063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10941.431553] Lustre: Skipped 1 previous similar message [10941.436617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10941.443948] Lustre: Skipped 2 previous similar messages [10943.232468] Lustre: server umount lustre-MDT0000 complete [10947.044489] LustreError: 366009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.070190] LustreError: 366009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10947.294231] LustreError: 368501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059791 with bad export cookie 7428988576046905270 [10947.300286] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10947.306658] LustreError: 368501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10947.760451] Lustre: server umount lustre-MDT0001 complete [10957.914417] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [10960.307521] Key type lgssc unregistered [10960.553813] LNet: 369540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10960.564259] LNetError: 369540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10960.592141] LNet: Removed LNI 192.168.203.156@tcp [10961.510654] Key type .llcrypt unregistered [10961.517083] Key type ._llcrypt unregistered [10986.128548] Key type ._llcrypt registered [10986.132093] Key type .llcrypt registered [10986.254781] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [10987.171421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10987.314162] alg: No test for adler32 (adler32-zlib) [10988.476398] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [10988.722088] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [10990.391711] Key type lgssc registered [10991.515722] Lustre: Echo OBD driver; http://www.lustre.org/ [10998.500326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10998.516561] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10998.523250] LustreError: 370976:0:(osd_handler.c:8543:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10998.547675] LustreError: 370976:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10998.556033] LustreError: 370976:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10998.572380] LustreError: 370976:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10998.579709] LustreError: 370976:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11006.605846] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11006.610775] LustreError: 371373:0:(osd_handler.c:8543:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11006.634945] LustreError: 371373:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11006.652997] LustreError: 371373:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11006.662754] LustreError: 371373:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11006.669853] LustreError: 371373:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11025.839683] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11028.663867] Key type lgssc unregistered [11028.978549] LNet: 372376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11028.981793] LNetError: 372376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11029.017462] LNet: Removed LNI 192.168.203.156@tcp [11029.892849] Key type .llcrypt unregistered [11029.896754] Key type ._llcrypt unregistered [11047.398704] Key type ._llcrypt registered [11047.402721] Key type .llcrypt registered [11047.524949] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [11062.696718] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11063.617886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11063.702326] alg: No test for adler32 (adler32-zlib) [11064.855533] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [11065.163934] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11066.855235] Key type lgssc registered [11067.828939] Lustre: Echo OBD driver; http://www.lustre.org/ [11102.524327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11102.551945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11103.775313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11103.804522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11103.873946] Lustre: lustre-MDT0000: new disk, initializing [11103.946599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11103.973635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11108.604901] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11119.456982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.592135] Lustre: 375531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11119.664880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11119.669952] Lustre: Skipped 1 previous similar message [11119.749273] Lustre: lustre-MDT0001: new disk, initializing [11119.881521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11119.921982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11119.942133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11124.552055] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11133.346894] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11139.292263] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11145.907080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11146.196459] Lustre: lustre-OST0000: new disk, initializing [11146.200413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11146.207792] Lustre: 377497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11146.284148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11152.376321] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11152.488286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11152.503287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11152.678156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11161.709706] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11166.702996] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11167.008882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11171.767796] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11171.955066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11178.477390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11178.480549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11178.493035] LustreError: Skipped 1 previous similar message [11178.498874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11178.513630] Lustre: Skipped 1 previous similar message [11178.534266] Lustre: Skipped 1 previous similar message [11179.874443] Lustre: server umount lustre-OST0000 complete [11186.656625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11186.666222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.688030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11188.705846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11188.705933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11188.727747] Lustre: Skipped 1 previous similar message [11190.034887] Lustre: server umount lustre-MDT0000 complete [11194.044255] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11194.044652] LustreError: 375520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060037 with bad export cookie 17021610706856531408 [11194.059453] LustreError: 375520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11194.352337] Lustre: server umount lustre-MDT0001 complete [11203.558915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11205.203745] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:07:27 (1789060047) [11211.386577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11211.823981] LustreError: 379960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11211.849749] LustreError: 379960:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11211.910906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11216.362708] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11217.384761] LustreError: 379961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11221.476808] LustreError: 379960:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11226.385113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11226.593805] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11226.744294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11230.859321] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11237.302623] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11241.903580] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11247.704977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11248.031981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11254.061846] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11261.148773] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11266.976322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11267.124873] Lustre: lustre-OST0001: new disk, initializing [11267.129824] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11267.137668] Lustre: 382793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11267.207303] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11273.070852] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11276.956997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11276.971930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11277.111266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11283.043609] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11290.984453] Lustre: lustre-OST0001: Not available for connect from 192.168.203.56@tcp (stopping) [11292.642940] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11292.643314] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11292.654447] LustreError: Skipped 1 previous similar message [11292.666543] Lustre: Skipped 1 previous similar message [11293.752445] Lustre: server umount lustre-OST0001 complete [11296.053607] LustreError: 381782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.885446] LustreError: 381781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.908384] LustreError: 381781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11312.611693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11312.618217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11312.639767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11312.648427] Lustre: Skipped 3 previous similar messages [11313.133950] LustreError: 381783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.147574] LustreError: 381783:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11323.871136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11323.999786] Lustre: server umount lustre-OST0000 complete [11328.483839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11328.497296] Lustre: Skipped 2 previous similar messages [11333.443134] Lustre: server umount lustre-MDT0000 complete [11333.604975] LustreError: 379961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.630986] LustreError: 379961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11336.517977] LustreError: 379941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060180 with bad export cookie 17021610706856532290 [11336.524541] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11336.531692] LustreError: 379941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11342.964029] Lustre: server umount lustre-MDT0001 complete [11351.765941] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11354.535476] Key type lgssc unregistered [11354.869917] LNet: 384784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11354.878276] LNetError: 384784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11354.910042] LNet: Removed LNI 192.168.203.156@tcp [11355.751202] Key type .llcrypt unregistered [11355.759951] Key type ._llcrypt unregistered [11409.499686] Key type ._llcrypt registered [11409.502935] Key type .llcrypt registered [11409.615787] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11411.451329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11411.562190] alg: No test for adler32 (adler32-zlib) [11412.703659] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [11413.101788] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11414.799215] Key type lgssc registered [11416.378116] Lustre: Echo OBD driver; http://www.lustre.org/ [11424.762491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11424.776033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11426.149453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11426.187573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11426.446251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11431.555296] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11441.139365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.258940] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11441.318622] Lustre: 387581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11441.640260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11446.390730] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11454.736243] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11460.626989] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11463.137053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.145965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11463.160021] Lustre: Skipped 1 previous similar message [11463.179126] Lustre: Skipped 1 previous similar message [11468.257190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11468.858247] Lustre: server umount lustre-MDT0000 complete [11472.314125] LustreError: 387572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060316 with bad export cookie 11443410755860443206 [11472.323305] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11472.332461] LustreError: 387572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11472.660861] Lustre: server umount lustre-MDT0001 complete [11481.230806] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:12:03 (1789060323) [11487.665428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11488.178087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11492.679175] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11493.349512] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11493.377095] LustreError: 390093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11498.471719] LustreError: 390092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.411641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11501.836049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11506.471444] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11513.331657] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11518.683554] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11524.983310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11525.137940] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11525.258194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11530.948616] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11536.811700] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11543.075379] Lustre: Permanently deactivating lustre-OST0000 [11543.091642] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11550.766927] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11551.763133] LustreError: 393077:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11551.771231] LustreError: 393077:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11551.777401] LustreError: 393077:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11554.364452] LustreError: 393126:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11554.370859] LustreError: 393126:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11562.449530] Lustre: server umount lustre-OST0000 complete [11564.868729] LustreError: 393376:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11564.873717] LustreError: 393376:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11568.609174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11568.622428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11568.639176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11568.646560] Lustre: Skipped 1 previous similar message [11569.635632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.754079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11574.759297] Lustre: Skipped 3 previous similar messages [11576.879514] Lustre: server umount lustre-MDT0000 complete [11579.876293] LustreError: 390828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11579.897647] LustreError: 390828:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11580.006901] LustreError: 390074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060423 with bad export cookie 11443410755860443787 [11580.012528] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11580.017990] LustreError: 390074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11589.866661] Lustre: server umount lustre-MDT0001 complete [11599.480496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.085982] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11606.947886] LustreError: 394536:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11613.345844] Lustre: 394875:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.156@tcp [11615.564991] Lustre: server umount MGS complete [11628.211370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11628.777155] LustreError: 395600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.818326] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11628.829370] Lustre: Skipped 1 previous similar message [11628.920522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11633.607757] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11634.145396] LustreError: 395601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11642.110811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11642.434222] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11646.672358] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11653.576421] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11659.766796] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11661.594111] Lustre: Permanently reactivating lustre-OST0000 [11661.609972] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11661.614390] Lustre: Skipped 2 previous similar messages [11689.448640] LustreError: 397379:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11697.510916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11697.841741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11697.845808] Lustre: Skipped 1 previous similar message [11704.060195] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11711.556433] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11718.667237] Lustre: server umount lustre-OST0000 complete [11724.257426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11724.263573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11724.280322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11732.038513] Lustre: server umount lustre-MDT0000 complete [11735.750318] LustreError: 395581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060579 with bad export cookie 11443410755860444837 [11735.763570] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11735.765254] LustreError: 395581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11745.467117] Lustre: server umount lustre-MDT0001 complete [11755.936933] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [11758.494894] Key type lgssc unregistered [11758.769960] LNet: 400013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11758.779924] LNetError: 400013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11758.795877] LNet: Removed LNI 192.168.203.156@tcp [11759.786143] Key type .llcrypt unregistered [11759.789963] Key type ._llcrypt unregistered [11777.420976] Key type ._llcrypt registered [11777.422386] Key type .llcrypt registered [11777.512362] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [11791.828604] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [11792.980533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11793.062308] alg: No test for adler32 (adler32-zlib) [11794.082704] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [11794.285955] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [11795.967190] Key type lgssc registered [11797.196718] Lustre: Echo OBD driver; http://www.lustre.org/ [11830.910806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11830.925923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.179636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11832.217716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11832.284806] Lustre: lustre-MDT0000: new disk, initializing [11832.364308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11832.376055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11836.527926] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11846.148353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11846.269640] Lustre: 403165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11846.343720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11846.354620] Lustre: Skipped 1 previous similar message [11846.489738] Lustre: lustre-MDT0001: new disk, initializing [11846.598568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11846.639383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11846.653310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11850.764521] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11859.976605] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11865.458263] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11871.826389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11872.166677] Lustre: lustre-OST0000: new disk, initializing [11872.170502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11872.178271] Lustre: 405132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11872.264655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11878.296836] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11882.050372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11882.065803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11882.238141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11889.445705] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11894.118597] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11894.393247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11899.142401] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11899.398777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11902.944776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11902.957325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11902.961263] Lustre: Skipped 1 previous similar message [11902.969224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11902.975908] Lustre: Skipped 1 previous similar message [11907.635969] Lustre: server umount lustre-OST0000 complete [11913.185098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11913.192268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11913.195182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11913.216226] Lustre: Skipped 1 previous similar message [11917.483683] Lustre: server umount lustre-MDT0000 complete [11920.944580] LustreError: 403159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060764 with bad export cookie 12935181803626241620 [11920.950718] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11920.962287] LustreError: 403159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11921.266391] Lustre: server umount lustre-MDT0001 complete [11929.640933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11931.287347] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:19:33 (1789060773) [11937.828622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.193519] LustreError: 407591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.206398] LustreError: 407591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11938.257593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11942.445411] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11943.411507] LustreError: 407592:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11948.514487] LustreError: 407591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.683942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11951.176103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11955.194308] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11962.890533] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11968.130533] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11974.313440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11974.619411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11979.859411] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [11988.027753] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11989.924546] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11992.512821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11999.208349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11999.221686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.234058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11999.234165] Lustre: Skipped 1 previous similar message [11999.280495] Lustre: Skipped 1 previous similar message [12004.093447] Lustre: server umount lustre-OST0000 complete [12009.441450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.459574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.467308] Lustre: Skipped 2 previous similar messages [12014.562740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.574936] Lustre: Skipped 1 previous similar message [12022.239598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12022.600089] Lustre: server umount lustre-MDT0000 complete [12024.811165] LustreError: 408319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12024.842244] LustreError: 408319:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12026.805283] LustreError: 410276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060870 with bad export cookie 12935181803626242502 [12026.808857] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12026.817198] LustreError: 410276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12027.339119] Lustre: server umount lustre-MDT0001 complete [12040.424381] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12044.142597] Key type lgssc unregistered [12044.581726] LNet: 411318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12044.601951] LNetError: 411318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12045.670711] LNet: Removed LNI 192.168.203.156@tcp [12046.670793] Key type .llcrypt unregistered [12046.675696] Key type ._llcrypt unregistered [12069.902933] Key type ._llcrypt registered [12069.905719] Key type .llcrypt registered [12070.063064] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12070.484523] Key type .llcrypt unregistered [12070.489523] Key type ._llcrypt unregistered [12087.217738] Key type ._llcrypt registered [12087.225766] Key type .llcrypt registered [12087.365421] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12088.921232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12088.987115] alg: No test for adler32 (adler32-zlib) [12090.440503] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [12090.880612] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12092.647676] Key type lgssc registered [12094.428572] Lustre: Echo OBD driver; http://www.lustre.org/ [12104.436592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12104.468962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12106.083673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12110.956554] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12115.426210] LustreError: 413522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12115.453098] LustreError: 413522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12118.370172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.712498] LustreError: 413519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12118.733966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12118.744708] LustreError: 413519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12124.132312] LustreError: 413522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.159064] LustreError: 413522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12124.936870] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12129.263897] LustreError: 413518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12129.300487] LustreError: 413518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12130.291444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12132.338481] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12135.402029] LustreError: 413517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.432944] LustreError: 413517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12139.392307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.879485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12144.129256] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12154.034568] Lustre: server umount lustre-OST0000 complete [12159.975879] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12159.986747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12159.996063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.618664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.623538] Lustre: Skipped 2 previous similar messages [12167.852747] Lustre: server umount lustre-MDT0000 complete [12170.725122] LustreError: 413518:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.741245] LustreError: 413518:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12171.606980] LustreError: 413504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061015 with bad export cookie 8880535045935924493 [12171.609356] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12171.622199] LustreError: 413504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12182.141552] Lustre: server umount lustre-MDT0001 complete [12192.622063] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12195.353201] Key type lgssc unregistered [12195.592967] LNet: 416672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12195.606414] LNetError: 416672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12195.620281] LNet: Removed LNI 192.168.203.156@tcp [12196.421294] Key type .llcrypt unregistered [12196.423319] Key type ._llcrypt unregistered [12219.684288] Key type ._llcrypt registered [12219.689464] Key type .llcrypt registered [12219.756956] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12220.428716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12220.450544] alg: No test for adler32 (adler32-zlib) [12221.459468] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [12221.666444] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12223.320093] Key type lgssc registered [12224.256789] Lustre: Echo OBD driver; http://www.lustre.org/ [12231.853619] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12231.893792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12247.522504] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12252.640052] LustreError: 418059:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.156@tcp: failed processing log, type 4: rc = -110 [12279.263327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.420213] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12292.926554] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12299.907759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.359824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12300.453806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12304.693553] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12305.379569] LustreError: 418085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12305.394721] LustreError: 418085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12310.498169] LustreError: 418084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.334597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12312.621168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12316.811426] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12324.100294] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12329.721238] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.150806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12338.161309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12338.179829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12339.147351] Lustre: server umount lustre-OST0000 complete [12343.266203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12343.276133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12343.304048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12344.297328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.311068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12344.323239] Lustre: Skipped 1 previous similar message [12349.408837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12349.419469] Lustre: Skipped 1 previous similar message [12352.584929] Lustre: server umount lustre-MDT0000 complete [12354.529073] LustreError: 419100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12354.555591] LustreError: 419100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12356.243295] LustreError: 418065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061200 with bad export cookie 11565675495498055465 [12356.246948] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12356.252490] LustreError: 418065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12356.607541] Lustre: server umount lustre-MDT0001 complete [12367.419615] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12370.044414] Key type lgssc unregistered [12370.321655] LNet: 421628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12370.330173] LNetError: 421628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12370.351756] LNet: Removed LNI 192.168.203.156@tcp [12371.093396] Key type .llcrypt unregistered [12371.095180] Key type ._llcrypt unregistered [12394.250863] Key type ._llcrypt registered [12394.253538] Key type .llcrypt registered [12394.402821] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12395.383671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12395.657526] alg: No test for adler32 (adler32-zlib) [12396.763823] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [12396.946802] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12398.591226] Key type lgssc registered [12399.881420] Lustre: Echo OBD driver; http://www.lustre.org/ [12406.935820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12406.973319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12408.426409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12412.145924] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12419.041710] LustreError: 423041:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12419.064239] LustreError: 423041:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12419.948291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.408251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12424.984523] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12432.477176] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12439.048229] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12446.428248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12446.743342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12453.820459] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12453.947952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12461.443327] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12466.656211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12466.666041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.694656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12469.594266] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.56@tcp (stopping) [12470.249053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12470.268397] Lustre: Skipped 1 previous similar message [12472.440370] Lustre: server umount lustre-MDT0000 complete [12474.630023] LustreError: 423021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.56@tcp arrived at 1789061318 with bad export cookie 17027916355746016092 [12474.639081] LustreError: 423021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12475.368156] LustreError: 424837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.384409] LustreError: 424837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12476.398462] LustreError: 424883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061320 with bad export cookie 17027916355746015203 [12476.414537] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12476.419248] LustreError: 424883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12480.481085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12480.482861] LustreError: 423040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12480.502472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12480.523716] LustreError: 423040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12480.565649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12480.580432] Lustre: Skipped 3 previous similar messages [12482.609452] Lustre: server umount lustre-OST0000 complete [12486.633044] LustreError: 423037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12486.655614] LustreError: 423037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12491.358643] Lustre: server umount lustre-MDT0001 complete [12503.846323] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12506.166312] Key type lgssc unregistered [12506.491790] LNet: 426726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12506.500743] LNetError: 426726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12506.526723] LNet: Removed LNI 192.168.203.156@tcp [12507.597117] Key type .llcrypt unregistered [12507.602458] Key type ._llcrypt unregistered [12534.208628] Key type ._llcrypt registered [12534.210315] Key type .llcrypt registered [12534.299917] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12535.511256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12535.630823] alg: No test for adler32 (adler32-zlib) [12536.873702] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [12537.145543] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12538.815527] Key type lgssc registered [12540.135845] Lustre: Echo OBD driver; http://www.lustre.org/ [12547.705649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12547.731849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.076237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.204530] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12558.310657] LustreError: 428140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12558.321404] LustreError: 428140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12561.566691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12561.952374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12566.637813] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12573.751281] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12579.225944] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12585.250563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12585.615206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12592.161701] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12593.861078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12599.675637] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12604.386218] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.410531] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12604.421427] Lustre: Skipped 1 previous similar message [12607.322426] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [12609.514404] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12609.527315] Lustre: Skipped 1 previous similar message [12610.702683] Lustre: server umount lustre-MDT0001 complete [12612.446084] LustreError: 429573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12614.647516] LustreError: 428136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12614.661190] LustreError: 428136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12619.749202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12619.751600] LustreError: 428869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12619.755886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12619.755896] Lustre: Skipped 1 previous similar message [12619.758035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12622.765290] Lustre: server umount lustre-OST0000 complete [12624.868248] LustreError: 428141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12626.844280] Lustre: server umount lustre-MDT0000 complete [12639.157068] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12642.024820] Key type lgssc unregistered [12642.302830] LNet: 431823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12642.314706] LNetError: 431823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12642.339144] LNet: Removed LNI 192.168.203.156@tcp [12643.272519] Key type .llcrypt unregistered [12643.275723] Key type ._llcrypt unregistered [12667.756277] Key type ._llcrypt registered [12667.761257] Key type .llcrypt registered [12667.884647] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [12682.549710] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12683.426878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12683.445744] alg: No test for adler32 (adler32-zlib) [12684.581874] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [12684.836851] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12686.487188] Key type lgssc registered [12687.641036] Lustre: Echo OBD driver; http://www.lustre.org/ [12723.305166] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12737.969295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12737.989181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12739.269873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12739.310507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12739.394957] Lustre: lustre-MDT0000: new disk, initializing [12739.489143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12739.502241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12743.239959] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12757.102523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12757.413989] Lustre: lustre-OST0000: new disk, initializing [12757.417986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12757.419993] Lustre: Skipped 1 previous similar message [12757.424369] Lustre: 436854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12757.523089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12762.795717] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12763.249942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12763.269351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12763.312116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12772.082297] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12788.245042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12788.462292] Lustre: 435835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12788.593324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12788.733146] Lustre: lustre-MDT0001: new disk, initializing [12788.864647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12788.928206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12788.942128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12794.565635] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [12804.298869] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12804.533157] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12810.663895] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12810.999829] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12818.284525] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [12819.937864] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12819.952230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12819.957216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12819.970947] Lustre: Skipped 1 previous similar message [12823.353261] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.56@tcp (stopping) [12823.372516] Lustre: Skipped 1 previous similar message [12823.856496] Lustre: server umount lustre-MDT0001 complete [12830.184729] LustreError: 435845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12830.216072] LustreError: 435845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12835.308721] LustreError: 435849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12835.333264] LustreError: 435849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12840.438047] LustreError: 435848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.468880] LustreError: 435848:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12845.024958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12845.045691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.070399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12845.540373] LustreError: 435849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.659443] LustreError: 435848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.661835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12857.311234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12857.499300] Lustre: server umount lustre-OST0000 complete [12860.901274] LustreError: 435848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12860.915056] LustreError: 435848:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12876.255155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12876.702398] Lustre: server umount lustre-MDT0000 complete [12894.635845] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12897.459393] Key type lgssc unregistered [12897.739829] LNet: 440604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12897.755755] LNetError: 440604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12897.773785] LNet: Removed LNI 192.168.203.156@tcp [12898.760911] Key type .llcrypt unregistered [12898.768889] Key type ._llcrypt unregistered [12936.625177] Key type ._llcrypt registered [12936.627851] Key type .llcrypt registered [12936.717552] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [12936.977452] Key type .llcrypt unregistered [12936.982362] Key type ._llcrypt unregistered [12952.384799] Key type ._llcrypt registered [12952.387499] Key type .llcrypt registered [12952.481531] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [12953.471820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12953.593443] alg: No test for adler32 (adler32-zlib) [12954.699890] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [12954.951629] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [12956.656377] Key type lgssc registered [12958.270360] Lustre: Echo OBD driver; http://www.lustre.org/ [12985.457812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12985.477843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12986.832781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12986.874911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12987.009995] Lustre: lustre-MDT0000: new disk, initializing [12987.120147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12987.164549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12992.557381] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13003.576630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.763930] Lustre: 443974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13003.821555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13003.831369] Lustre: Skipped 1 previous similar message [13003.943452] Lustre: lustre-MDT0001: new disk, initializing [13004.055589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13004.102623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13004.118440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13009.284581] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13019.155591] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13024.747816] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13031.384946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13031.861282] Lustre: lustre-OST0000: new disk, initializing [13031.871815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13031.880649] Lustre: 445941:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13032.009723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13037.685665] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13040.185923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13040.194680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13040.272953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13046.696606] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13055.457632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13055.470954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13055.502566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13055.974870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13055.997105] Lustre: Skipped 1 previous similar message [13056.005771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13056.011418] Lustre: Skipped 1 previous similar message [13059.195578] Lustre: server umount lustre-MDT0000 complete [13062.458918] LustreError: 443966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061906 with bad export cookie 4427010332668592361 [13062.469825] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13062.471045] LustreError: 443966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13062.722984] Lustre: server umount lustre-MDT0001 complete [13076.702906] Lustre: server umount lustre-OST0000 complete [13086.484696] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:38:48 (1789061928) [13093.759851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13094.149704] LustreError: 447968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13094.168287] LustreError: 447968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13094.219415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13098.628035] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13099.492138] LustreError: 447969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13104.616417] LustreError: 447968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13107.390858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13107.798554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13111.740289] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13118.266153] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13123.352474] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13142.774697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13143.063879] Lustre: Found index 0 for lustre-OST0000, updating log [13143.206202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13149.854817] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13156.330750] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13165.752839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13173.216978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13173.237862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13173.258241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13177.767341] Lustre: server umount lustre-OST0000 complete [13194.720371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13194.731149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13194.742263] Lustre: Skipped 1 previous similar message [13194.746210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13194.749185] Lustre: Skipped 3 previous similar messages [13199.873432] Lustre: server umount lustre-MDT0000 complete [13200.361915] LustreError: 447969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13200.379855] LustreError: 447969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13203.389040] LustreError: 447950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062047 with bad export cookie 4427010332668593754 [13203.402207] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13203.407766] LustreError: 447950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13203.806401] Lustre: server umount lustre-MDT0001 complete [13205.856499] LNet: 1 local NIs in recovery (showing 1): 192.168.203.156@tcp [13212.388936] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:40:54 (1789062054) [13836.631259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13837.220668] LustreError: 454400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.241259] LustreError: 454400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13837.345344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13841.897696] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13842.401801] LustreError: 454401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13847.538948] LustreError: 454400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13851.580826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13852.092633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13857.070804] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13865.551465] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13872.266329] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13893.056525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13893.535409] Lustre: Found index 0 for lustre-OST0000, updating log [13894.051854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13895.075106] LNet: 1 local NIs in recovery (showing 1): 192.168.203.156@tcp [13901.471568] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [13910.106646] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13913.332333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13913.353548] Lustre: Skipped 1 previous similar message [13913.369834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13913.374233] Lustre: Skipped 2 previous similar messages [13918.575223] Lustre: server umount lustre-OST0000 complete [13923.365266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13923.366382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13923.383157] Lustre: Skipped 1 previous similar message [13923.401489] Lustre: Skipped 1 previous similar message [13924.452125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13924.460761] Lustre: Skipped 1 previous similar message [13926.497711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13926.510304] Lustre: Skipped 3 previous similar messages [13928.726500] Lustre: server umount lustre-MDT0000 complete [13929.569124] LustreError: 454400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13929.615198] LustreError: 454400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13933.078336] LustreError: 454380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062776 with bad export cookie 4427010332668594839 [13933.078920] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13933.096985] LustreError: 454380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13933.691759] Lustre: server umount lustre-MDT0001 complete [13954.528419] LNet: 1 local NIs in recovery (showing 1): 192.168.203.156@tcp [13966.265163] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [13976.756719] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14018.220808] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:54:20 (1789062860) [14039.646474] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14043.204200] Key type lgssc unregistered [14043.588312] LNet: 461012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14043.600778] LNetError: 461012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14157.288339] LNet: Removed LNI 192.168.203.156@tcp [14158.281235] Key type .llcrypt unregistered [14158.284234] Key type ._llcrypt unregistered [14177.161272] Key type ._llcrypt registered [14177.164106] Key type .llcrypt registered [14177.358370] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14178.966364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14179.015508] alg: No test for adler32 (adler32-zlib) [14180.559934] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [14180.926121] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14182.736335] Key type lgssc registered [14184.328675] Lustre: Echo OBD driver; http://www.lustre.org/ [14191.018221] LNet: Added LNI 192.168.203.156@tcp100 [8/256/0/180] [14192.097460] LNet: Added LNI 192.168.203.156@tcp101 [8/256/0/180] [14193.310349] LNet: Added LNI 192.168.203.156@tcp102 [8/256/0/180] [14194.470392] LNet: Added LNI 192.168.203.156@tcp103 [8/256/0/180] [14195.465547] LNet: Added LNI 192.168.203.156@tcp104 [8/256/0/180] [14196.441951] LNet: Added LNI 192.168.203.156@tcp105 [8/256/0/180] [14197.301165] LNet: Added LNI 192.168.203.156@tcp106 [8/256/0/180] [14198.437608] LNet: Added LNI 192.168.203.156@tcp107 [8/256/0/180] [14199.857549] LNet: Added LNI 192.168.203.156@tcp108 [8/256/0/180] [14201.007988] LNet: Added LNI 192.168.203.156@tcp109 [8/256/0/180] [14202.157396] LNet: Added LNI 192.168.203.156@tcp110 [8/256/0/180] [14203.182627] LNet: Added LNI 192.168.203.156@tcp111 [8/256/0/180] [14204.198524] LNet: Added LNI 192.168.203.156@tcp112 [8/256/0/180] [14205.244860] LNet: Added LNI 192.168.203.156@tcp113 [8/256/0/180] [14206.430453] LNet: Added LNI 192.168.203.156@tcp114 [8/256/0/180] [14207.691335] LNet: Added LNI 192.168.203.156@tcp115 [8/256/0/180] [14208.626310] LNet: Added LNI 192.168.203.156@tcp116 [8/256/0/180] [14209.729553] LNet: Added LNI 192.168.203.156@tcp117 [8/256/0/180] [14211.101269] LNet: Added LNI 192.168.203.156@tcp118 [8/256/0/180] [14212.122535] LNet: Added LNI 192.168.203.156@tcp119 [8/256/0/180] [14213.252181] LNet: Added LNI 192.168.203.156@tcp120 [8/256/0/180] [14214.606339] LNet: Added LNI 192.168.203.156@tcp121 [8/256/0/180] [14215.804510] LNet: Added LNI 192.168.203.156@tcp122 [8/256/0/180] [14217.377237] LNet: Added LNI 192.168.203.156@tcp123 [8/256/0/180] [14219.037777] LNet: Added LNI 192.168.203.156@tcp124 [8/256/0/180] [14220.283971] LNet: Added LNI 192.168.203.156@tcp125 [8/256/0/180] [14221.346589] LNet: Added LNI 192.168.203.156@tcp126 [8/256/0/180] [14222.405892] LNet: Added LNI 192.168.203.156@tcp127 [8/256/0/180] [14223.395115] LNet: Added LNI 192.168.203.156@tcp128 [8/256/0/180] [14224.521675] LNet: Added LNI 192.168.203.156@tcp129 [8/256/0/180] [14225.802096] LNet: Added LNI 192.168.203.156@tcp130 [8/256/0/180] [14226.757358] LNet: Added LNI 192.168.203.156@tcp131 [8/256/0/180] [14227.737921] LNet: Added LNI 192.168.203.156@tcp132 [8/256/0/180] [14228.808864] LNet: Added LNI 192.168.203.156@tcp133 [8/256/0/180] [14229.735541] LNet: Added LNI 192.168.203.156@tcp134 [8/256/0/180] [14230.873597] LNet: Added LNI 192.168.203.156@tcp135 [8/256/0/180] [14239.070821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14239.127947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14240.318976] Lustre: 464067:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [14240.449594] LNetError: 461523:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14240.471373] LNetError: Protocol error connecting to 192.168.203.156@tcp119 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14240.483077] LNetError: Skipped 1 previous similar message [14240.657785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14240.711086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14240.828492] Lustre: lustre-MDT0000: new disk, initializing [14240.956119] Lustre: 464123:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [14240.979244] Lustre: 464123:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14241.002248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14241.036659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14241.385031] LNetError: 464091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14241.388545] LNet: 9 local NIs in recovery (showing 9): 192.168.203.156@tcp122, 192.168.203.156@tcp123, 192.168.203.156@tcp124, 192.168.203.156@tcp125, 192.168.203.156@tcp119, 192.168.203.156@tcp127, 192.168.203.156@tcp128, 192.168.203.156@tcp129, 192.168.203.156@tcp126 [14241.404616] LNetError: 464091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14241.427668] LNetError: Protocol error connecting to 192.168.203.156@tcp100 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14241.471446] LNetError: Skipped 12 previous similar messages [14243.488426] LNetError: 464096:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14243.513732] LNetError: 464096:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14243.530637] LNetError: Protocol error connecting to 192.168.203.156@tcp129 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14246.546294] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14247.584431] LNetError: 461522:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14247.601316] LNetError: Protocol error connecting to 192.168.203.156@tcp129 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14255.784931] LNetError: 461523:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14255.806147] LNetError: Protocol error connecting to 192.168.203.156@tcp129 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14257.863678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14258.060380] Lustre: 464083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14258.122404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14258.129946] Lustre: Skipped 1 previous similar message [14258.251505] Lustre: lustre-MDT0001: new disk, initializing [14258.307392] Lustre: 464887:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [14258.313412] Lustre: 464887:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14258.425827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14258.466625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14258.481591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14263.250287] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14264.869859] Lustre: 465218:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [14264.877155] Lustre: 465218:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14271.145087] LNetError: 461524:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14271.189416] LNetError: Protocol error connecting to 192.168.203.156@tcp125 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14273.448659] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14281.834359] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14287.523429] LNetError: 464091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14287.539541] LNetError: 464091:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14287.550473] LNetError: Protocol error connecting to 192.168.203.156@tcp128 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14287.565279] LNetError: Skipped 7 previous similar messages [14289.154989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14289.618537] Lustre: lustre-OST0000: new disk, initializing [14289.625223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14289.633586] Lustre: 466060:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14289.770459] Lustre: 466065:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [14289.783938] Lustre: 466065:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14289.858904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14296.695027] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14299.195864] Lustre: 466517:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.156@tcp: can't add NID 192.168.203.156@tcp130: rc = -75 [14299.223796] Lustre: 466517:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14299.297991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14299.309737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14299.490282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14306.847456] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14320.106470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14320.108204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14320.113701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14320.118824] LustreError: Skipped 1 previous similar message [14320.148697] Lustre: Skipped 1 previous similar message [14320.294989] LNetError: 461522:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.156 [14320.318570] LNetError: 461522:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [14320.333540] LNetError: Protocol error connecting to 192.168.203.156@tcp127 at host 192.168.203.156:988: is it running a compatible version of Lustre? [14320.341653] LNetError: Skipped 16 previous similar messages [14321.999978] Lustre: server umount lustre-OST0000 complete [14330.340164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14330.356143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14330.365520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14330.368636] Lustre: Skipped 1 previous similar message [14330.389445] Lustre: Skipped 2 previous similar messages [14331.940600] Lustre: server umount lustre-MDT0000 complete [14336.062317] LustreError: 464077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063179 with bad export cookie 1751088637800702675 [14336.066198] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14336.074221] LustreError: 464077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14336.591441] Lustre: server umount lustre-MDT0001 complete [14350.317725] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14353.937226] Key type lgssc unregistered [14354.565267] LNet: 467975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14354.574985] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14354.595173] LNet: Removed LNI 192.168.203.156@tcp [14355.615148] LNet: Removed LNI 192.168.203.156@tcp100 [14355.617156] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14355.631188] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14355.644719] LNet: Removed LNI 192.168.203.156@tcp101 [14356.703375] LNet: Removed LNI 192.168.203.156@tcp102 [14356.705581] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14356.725922] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14356.755045] LNet: Removed LNI 192.168.203.156@tcp103 [14357.791177] LNet: Removed LNI 192.168.203.156@tcp104 [14358.816062] LNet: Removed LNI 192.168.203.156@tcp105 [14358.818216] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14358.867155] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14358.889607] LNet: Removed LNI 192.168.203.156@tcp106 [14359.968249] LNet: Removed LNI 192.168.203.156@tcp107 [14360.992333] LNet: Removed LNI 192.168.203.156@tcp108 [14362.015472] LNet: Removed LNI 192.168.203.156@tcp109 [14363.039666] LNet: Removed LNI 192.168.203.156@tcp110 [14363.058729] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14363.093693] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14363.121942] LNet: Removed LNI 192.168.203.156@tcp111 [14364.193749] LNet: Removed LNI 192.168.203.156@tcp112 [14365.215304] LNet: Removed LNI 192.168.203.156@tcp113 [14366.239230] LNet: Removed LNI 192.168.203.156@tcp114 [14367.263147] LNet: Removed LNI 192.168.203.156@tcp115 [14368.287664] LNet: Removed LNI 192.168.203.156@tcp116 [14369.311197] LNet: Removed LNI 192.168.203.156@tcp117 [14370.335984] LNet: Removed LNI 192.168.203.156@tcp118 [14371.359147] LNet: Removed LNI 192.168.203.156@tcp119 [14371.370168] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14371.399348] LNetError: 467975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14371.416276] LNet: Removed LNI 192.168.203.156@tcp120 [14372.448465] LNet: Removed LNI 192.168.203.156@tcp121 [14373.471171] LNet: Removed LNI 192.168.203.156@tcp122 [14374.495164] LNet: Removed LNI 192.168.203.156@tcp123 [14375.520294] LNet: Removed LNI 192.168.203.156@tcp124 [14376.544603] LNet: Removed LNI 192.168.203.156@tcp125 [14377.568498] LNet: Removed LNI 192.168.203.156@tcp126 [14378.591182] LNet: Removed LNI 192.168.203.156@tcp127 [14379.615167] LNet: Removed LNI 192.168.203.156@tcp128 [14380.639669] LNet: Removed LNI 192.168.203.156@tcp129 [14380.645214] LNet: Removed LNI 192.168.203.156@tcp130 [14380.659168] LNet: Removed LNI 192.168.203.156@tcp131 [14380.668939] LNet: Removed LNI 192.168.203.156@tcp132 [14380.682138] LNet: Removed LNI 192.168.203.156@tcp133 [14380.687377] LNet: Removed LNI 192.168.203.156@tcp134 [14380.712449] LNet: Removed LNI 192.168.203.156@tcp135 [14381.936191] Key type .llcrypt unregistered [14381.937967] Key type ._llcrypt unregistered [14409.359965] Key type ._llcrypt registered [14409.362583] Key type .llcrypt registered [14409.481876] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [14426.317334] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14427.992781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14428.041532] alg: No test for adler32 (adler32-zlib) [14429.253336] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [14429.577819] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14431.383209] Key type lgssc registered [14432.715266] Lustre: Echo OBD driver; http://www.lustre.org/ [14469.984988] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:01:51 (1789063311) [14490.100199] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [14493.611201] Key type lgssc unregistered [14493.898324] LNet: 472072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14493.905503] LNetError: 472072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14493.933255] LNet: Removed LNI 192.168.203.156@tcp [14494.982412] Key type .llcrypt unregistered [14494.989766] Key type ._llcrypt unregistered [14513.871065] Key type ._llcrypt registered [14513.875713] Key type .llcrypt registered [14514.013236] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14515.424815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14515.484989] alg: No test for adler32 (adler32-zlib) [14516.633844] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [14516.977453] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [14518.759302] Key type lgssc registered [14519.993442] Lustre: Echo OBD driver; http://www.lustre.org/ [14530.127984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14530.180952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14531.466892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14531.505985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14531.687328] Lustre: lustre-MDT0000: new disk, initializing [14531.792981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14531.812778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14536.894985] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14549.416493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14549.691903] Lustre: 473281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14549.809863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14549.826683] Lustre: Skipped 1 previous similar message [14550.011442] Lustre: lustre-MDT0001: new disk, initializing [14550.333421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14550.425266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14550.452841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14556.678571] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14568.558583] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14575.640458] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14583.992171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14584.400167] Lustre: lustre-OST0000: new disk, initializing [14584.407273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14584.419507] Lustre: 475250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14584.531423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14591.480942] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14593.660319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14593.678133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14593.797796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14601.407687] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14606.325321] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing lnet_if_list [14612.529590] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14612.576272] LNet: Added LNI 192.168.203.156@tcp10 [8/256/0/180] [14618.787699] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14618.817256] LNet: Added LNI 192.168.203.156@tcp11 [8/256/0/180] [14624.057423] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14624.106244] LNet: Added LNI 192.168.203.156@tcp12 [8/256/0/180] [14629.239915] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14629.282684] LNet: Added LNI 192.168.203.156@tcp13 [8/256/0/180] [14634.112257] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14634.157108] LNet: Added LNI 192.168.203.156@tcp14 [8/256/0/180] [14639.396817] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14639.441696] LNet: Added LNI 192.168.203.156@tcp15 [8/256/0/180] [14645.050218] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14645.097249] LNet: Added LNI 192.168.203.156@tcp16 [8/256/0/180] [14650.863890] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14650.905231] LNet: Added LNI 192.168.203.156@tcp17 [8/256/0/180] [14656.385845] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14656.421210] LNet: Added LNI 192.168.203.156@tcp18 [8/256/0/180] [14661.748815] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14661.777497] LNet: Added LNI 192.168.203.156@tcp19 [8/256/0/180] [14667.417951] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14667.453896] LNet: Added LNI 192.168.203.156@tcp20 [8/256/0/180] [14673.049688] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14673.090983] LNet: Added LNI 192.168.203.156@tcp21 [8/256/0/180] [14679.482853] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14679.528120] LNet: Added LNI 192.168.203.156@tcp22 [8/256/0/180] [14686.297069] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14686.326157] LNet: Added LNI 192.168.203.156@tcp23 [8/256/0/180] [14693.465276] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14693.498607] LNet: Added LNI 192.168.203.156@tcp24 [8/256/0/180] [14701.073500] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14701.129572] LNet: Added LNI 192.168.203.156@tcp25 [8/256/0/180] [14708.524740] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14708.547495] LNet: Added LNI 192.168.203.156@tcp26 [8/256/0/180] [14716.638949] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14716.706311] LNet: Added LNI 192.168.203.156@tcp27 [8/256/0/180] [14724.016708] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14724.034030] LNet: Added LNI 192.168.203.156@tcp28 [8/256/0/180] [14730.956700] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14730.988613] LNet: Added LNI 192.168.203.156@tcp29 [8/256/0/180] [14740.196231] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14740.257659] LNet: Added LNI 192.168.203.156@tcp30 [8/256/0/180] [14748.280268] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14748.345495] LNet: Added LNI 192.168.203.156@tcp31 [8/256/0/180] [14755.925169] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14755.968462] LNet: Added LNI 192.168.203.156@tcp32 [8/256/0/180] [14764.810927] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14764.858767] LNet: Added LNI 192.168.203.156@tcp33 [8/256/0/180] [14772.406132] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14772.439782] LNet: Added LNI 192.168.203.156@tcp34 [8/256/0/180] [14780.179632] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14780.237462] LNet: Added LNI 192.168.203.156@tcp35 [8/256/0/180] [14788.047374] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14788.114312] LNet: Added LNI 192.168.203.156@tcp36 [8/256/0/180] [14796.841867] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14796.917799] LNet: Added LNI 192.168.203.156@tcp37 [8/256/0/180] [14805.343776] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14805.382864] LNet: Added LNI 192.168.203.156@tcp38 [8/256/0/180] [14814.239937] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14814.269499] LNet: Added LNI 192.168.203.156@tcp39 [8/256/0/180] [14821.494054] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14821.520729] LNet: Added LNI 192.168.203.156@tcp40 [8/256/0/180] [14829.132608] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14829.163614] LNet: Added LNI 192.168.203.156@tcp41 [8/256/0/180] [14836.314925] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14836.346738] LNet: Added LNI 192.168.203.156@tcp42 [8/256/0/180] [14844.565609] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14844.586023] LNet: Added LNI 192.168.203.156@tcp43 [8/256/0/180] [14853.169969] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14853.211796] LNet: Added LNI 192.168.203.156@tcp44 [8/256/0/180] [14861.802618] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14861.855172] LNet: Added LNI 192.168.203.156@tcp45 [8/256/0/180] [14870.860577] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14870.904381] LNet: Added LNI 192.168.203.156@tcp46 [8/256/0/180] [14879.786571] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14879.828643] LNet: Added LNI 192.168.203.156@tcp47 [8/256/0/180] [14888.409987] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14888.451952] LNet: Added LNI 192.168.203.156@tcp48 [8/256/0/180] [14897.093943] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14897.120954] LNet: Added LNI 192.168.203.156@tcp49 [8/256/0/180] [14905.662369] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14905.681376] LNet: Added LNI 192.168.203.156@tcp50 [8/256/0/180] [14926.816340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14926.826972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14926.835776] Lustre: Skipped 2 previous similar messages [14926.852932] Lustre: Skipped 2 previous similar messages [14931.936234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14931.943651] Lustre: Skipped 2 previous similar messages [14932.466684] Lustre: server umount lustre-MDT0000 complete [14936.332268] LustreError: 473273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063780 with bad export cookie 15890137231039532928 [14936.333274] LustreError: 473272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063780 with bad export cookie 15890137231039532928 [14936.357160] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14936.416396] LustreError: 473272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14936.806142] Lustre: server umount lustre-MDT0001 complete [14952.226663] Lustre: 472553:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063780/real 1789063780] req@00000000b175b0c8 x1875968901958656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789063796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14952.269955] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14955.765409] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [14957.535095] Lustre: 472554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063785/real 1789063785] req@00000000f8260af9 x1875968901958784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789063801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14958.623188] Lustre: 472554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063786/real 1789063786] req@000000002e0083d1 x1875968901958912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789063802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14962.655416] Lustre: 472552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063790/real 1789063790] req@00000000bbe56874 x1875968901959168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789063806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14964.131606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14967.839263] Lustre: 472552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789063795/real 1789063795] req@00000000cade3609 x1875968901959296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789063811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14982.500127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14983.531057] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14988.360801] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [14988.785147] LustreError: 475242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14988.803472] LustreError: 475242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [14993.893841] LustreError: 475243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14997.991629] LustreError: 499497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14998.016683] LustreError: 499497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14998.176577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14998.721971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15004.014222] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15004.142699] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15011.949595] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15017.426663] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15038.949960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15038.953437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15038.982963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15039.973417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15041.888306] Lustre: server umount lustre-OST0000 complete [15049.697176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15049.711185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15049.736434] Lustre: Skipped 1 previous similar message [15049.745501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15049.766214] Lustre: Skipped 2 previous similar messages [15052.498416] Lustre: server umount lustre-MDT0000 complete [15055.334123] LustreError: 499500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15055.355359] LustreError: 499500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15057.136300] LustreError: 475249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063901 with bad export cookie 15890137231039535560 [15057.137190] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15057.156527] LustreError: 475249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15057.652744] Lustre: server umount lustre-MDT0001 complete [15072.049700] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [15075.565174] Key type lgssc unregistered [15076.134711] LNet: 502276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15076.159597] LNetError: 502276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15076.172573] LNet: Removed LNI 192.168.203.156@tcp [15076.176178] LNet: Removed LNI 192.168.203.156@tcp10 [15076.184774] LNet: Removed LNI 192.168.203.156@tcp11 [15076.189975] LNet: Removed LNI 192.168.203.156@tcp12 [15076.196294] LNet: Removed LNI 192.168.203.156@tcp13 [15076.203928] LNet: Removed LNI 192.168.203.156@tcp14 [15076.210812] LNet: Removed LNI 192.168.203.156@tcp15 [15076.216832] LNet: Removed LNI 192.168.203.156@tcp16 [15076.224798] LNet: Removed LNI 192.168.203.156@tcp17 [15076.238324] LNet: Removed LNI 192.168.203.156@tcp18 [15076.244413] LNet: Removed LNI 192.168.203.156@tcp19 [15076.249841] LNet: Removed LNI 192.168.203.156@tcp20 [15076.256453] LNet: Removed LNI 192.168.203.156@tcp21 [15076.264437] LNet: Removed LNI 192.168.203.156@tcp22 [15076.268758] LNet: Removed LNI 192.168.203.156@tcp23 [15076.286820] LNet: Removed LNI 192.168.203.156@tcp24 [15076.292536] LNet: Removed LNI 192.168.203.156@tcp25 [15076.302151] LNet: Removed LNI 192.168.203.156@tcp26 [15076.310721] LNet: Removed LNI 192.168.203.156@tcp27 [15076.317756] LNet: Removed LNI 192.168.203.156@tcp28 [15076.325804] LNet: Removed LNI 192.168.203.156@tcp29 [15076.330179] LNet: Removed LNI 192.168.203.156@tcp30 [15076.333463] LNet: Removed LNI 192.168.203.156@tcp31 [15076.337116] LNet: Removed LNI 192.168.203.156@tcp32 [15076.341381] LNet: Removed LNI 192.168.203.156@tcp33 [15076.344738] LNet: Removed LNI 192.168.203.156@tcp34 [15076.351325] LNet: Removed LNI 192.168.203.156@tcp35 [15076.359135] LNet: Removed LNI 192.168.203.156@tcp36 [15076.367815] LNet: Removed LNI 192.168.203.156@tcp37 [15076.381612] LNet: Removed LNI 192.168.203.156@tcp38 [15076.387874] LNet: Removed LNI 192.168.203.156@tcp39 [15076.397832] LNet: Removed LNI 192.168.203.156@tcp40 [15076.400791] LNet: Removed LNI 192.168.203.156@tcp41 [15076.406295] LNet: Removed LNI 192.168.203.156@tcp42 [15076.409159] LNet: Removed LNI 192.168.203.156@tcp43 [15076.423322] LNet: Removed LNI 192.168.203.156@tcp44 [15076.429286] LNet: Removed LNI 192.168.203.156@tcp45 [15076.432659] LNet: Removed LNI 192.168.203.156@tcp46 [15076.435953] LNet: Removed LNI 192.168.203.156@tcp47 [15076.438986] LNet: Removed LNI 192.168.203.156@tcp48 [15076.442112] LNet: Removed LNI 192.168.203.156@tcp49 [15076.451377] LNet: Removed LNI 192.168.203.156@tcp50 [15077.350177] Key type .llcrypt unregistered [15077.356371] Key type ._llcrypt unregistered [15099.898854] Key type ._llcrypt registered [15099.901222] Key type .llcrypt registered [15100.011271] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [15115.797198] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15118.025506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15118.054449] alg: No test for adler32 (adler32-zlib) [15119.162640] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [15119.417315] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [15121.120528] Key type lgssc registered [15122.513762] Lustre: Echo OBD driver; http://www.lustre.org/ [15159.273680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15159.320903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15160.642037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15160.688327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15160.854426] Lustre: lustre-MDT0000: new disk, initializing [15160.983399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15161.012371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15165.818706] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15176.141523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15176.299737] Lustre: 505474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15176.351676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15176.356846] Lustre: Skipped 1 previous similar message [15176.427067] Lustre: lustre-MDT0001: new disk, initializing [15176.529307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15176.569319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15176.586709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15181.256828] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15190.313411] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15195.506996] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15203.806364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15204.098513] Lustre: *** cfs_fail_loc=90f, val=0*** [15204.226648] Lustre: lustre-OST0000: new disk, initializing [15204.237674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15204.251521] Lustre: 507491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15204.380716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15210.121530] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15212.644725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15212.653843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15212.778774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15219.124583] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15228.385071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15228.393332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15228.401958] LustreError: Skipped 1 previous similar message [15228.406938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15228.435941] Lustre: Skipped 1 previous similar message [15233.054816] Lustre: server umount lustre-OST0000 complete [15238.113174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15238.130251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15238.159847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15238.167258] Lustre: Skipped 1 previous similar message [15243.614296] Lustre: server umount lustre-MDT0000 complete [15248.351301] LustreError: 505465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064092 with bad export cookie 11592835061019021939 [15248.352618] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15248.362093] LustreError: 505465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15248.958959] Lustre: server umount lustre-MDT0001 complete [15259.757584] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [15262.733931] Key type lgssc unregistered [15263.131610] LNet: 509393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15263.147077] LNetError: 509393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15263.177624] LNet: Removed LNI 192.168.203.156@tcp [15264.271672] Key type .llcrypt unregistered [15264.275215] Key type ._llcrypt unregistered [15293.161432] Key type ._llcrypt registered [15293.167075] Key type .llcrypt registered [15293.338801] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15294.975189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15295.185821] alg: No test for adler32 (adler32-zlib) [15296.330317] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [15296.567889] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [15298.232323] Key type lgssc registered [15299.486438] Lustre: Echo OBD driver; http://www.lustre.org/ [15310.095510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15310.134546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15311.840140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15316.749609] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15322.096744] LustreError: 510663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15322.129074] LustreError: 510663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15327.212993] LustreError: 510664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15327.436552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15328.188413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15333.788154] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15342.031640] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15349.023586] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15357.360240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15357.917676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15364.860737] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15372.860330] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15383.141340] loop0: detected capacity change from 0 to 204800000 [15383.412148] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15383.428900] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15383.452393] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15389.810067] loop0: detected capacity change from 0 to 204800000 [15390.039817] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15390.055287] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15390.076093] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15395.471780] loop0: detected capacity change from 0 to 204800000 [15401.171487] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15401.323374] Lustre: fs15246-OST0000: new disk, initializing [15401.326363] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15401.336741] Lustre: 514254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15401.387477] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15409.117576] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15418.482473] loop1: detected capacity change from 0 to 204800000 [15423.613655] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15423.766406] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15423.858955] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15423.974213] Lustre: fs15246-MDT0000: new disk, initializing [15424.156682] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15424.183962] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15424.276755] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15425.311739] Lustre: 515378:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fb1c80d1 x1875969719713152/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789064285 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15428.775822] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15430.370261] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15430.390918] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15430.427663] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15450.593450] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15450.609153] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15455.281575] Lustre: server umount fs15246-MDT0000 complete [15455.738852] LustreError: 510658:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15460.837390] LustreError: 511617:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15464.478933] Lustre: server umount fs15246-OST0000 complete [15476.194589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15476.200371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15476.226230] Lustre: Skipped 1 previous similar message [15476.243517] Lustre: Skipped 1 previous similar message [15481.320932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15486.436250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15486.453642] Lustre: Skipped 2 previous similar messages [15490.015593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15490.165659] Lustre: server umount lustre-OST0000 complete [15496.672765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15496.675524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15496.690750] Lustre: Skipped 1 previous similar message [15496.704016] Lustre: Skipped 1 previous similar message [15500.254688] Lustre: server umount lustre-MDT0000 complete [15501.792528] LustreError: 515352:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15501.815669] LustreError: 515352:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15504.245768] LustreError: 510642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064348 with bad export cookie 3750343679662443681 [15504.249902] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15504.265173] LustreError: 510642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15504.896468] Lustre: server umount lustre-MDT0001 complete [15516.230660] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [15519.062416] Key type lgssc unregistered [15519.321637] LNet: 518476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15519.336221] LNetError: 518476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15519.358275] LNet: Removed LNI 192.168.203.156@tcp [15520.294678] Key type .llcrypt unregistered [15520.297535] Key type ._llcrypt unregistered [15537.012749] Key type ._llcrypt registered [15537.015694] Key type .llcrypt registered [15537.115811] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15538.406857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15538.443082] alg: No test for adler32 (adler32-zlib) [15539.509616] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [15539.791879] LNet: Added LNI 192.168.203.156@tcp [8/256/0/180] [15541.479297] Key type lgssc registered [15542.501842] Lustre: Echo OBD driver; http://www.lustre.org/ [15577.024414] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [15587.652856] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15625.707238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15625.728611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15627.014464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15627.063126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15627.175783] Lustre: lustre-MDT0000: new disk, initializing [15627.273488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15627.305330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15632.015741] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15642.646572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15642.758449] Lustre: 522938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15642.916830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15642.920715] Lustre: Skipped 1 previous similar message [15643.000808] Lustre: lustre-MDT0001: new disk, initializing [15643.122032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15643.173714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15643.181955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15647.581714] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15656.443717] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15661.583122] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15667.327738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15667.530049] Lustre: lustre-OST0000: new disk, initializing [15667.535971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15667.542719] Lustre: 524907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15667.625219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15673.315566] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_default_debug -1 all [15674.947289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15674.970300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15675.137752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15682.578293] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15687.243140] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15687.478936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15692.774264] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15692.991320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15695.840492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15695.844462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15695.852714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15695.870100] Lustre: Skipped 1 previous similar message [15700.967938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15700.979967] Lustre: Skipped 2 previous similar messages [15706.084823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15706.092220] Lustre: Skipped 1 previous similar message [15709.151178] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15709.452556] Lustre: server umount lustre-OST0000 complete [15714.784186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15714.798151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15714.817409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15716.322766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15719.274895] Lustre: server umount lustre-MDT0000 complete [15721.449264] LustreError: 525370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15721.478237] LustreError: 525370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [15723.504095] LustreError: 524906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064567 with bad export cookie 5182263801922110496 [15723.505597] LustreError: MGC192.168.203.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15723.519427] LustreError: 524906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15723.982842] Lustre: server umount lustre-MDT0001 complete [15751.453845] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing set_hostid [15763.025641] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing load_modules_local [15799.601500] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15518 sec ========= 14:24:01 (1789064641) [15801.551994] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:24:03 (1789064643) === [15805.033471] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:24:06 (1789064646) === [15845.898689] Lustre: DEBUG MARKER: oleg356-server.virtnet: executing unload_modules_local [15848.858793] Key type lgssc unregistered [15849.239640] LNet: 531554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15849.260214] LNetError: 531554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15849.273182] LNet: Removed LNI 192.168.203.156@tcp [15850.165802] Key type .llcrypt unregistered [15850.168450] Key type ._llcrypt unregistered