[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 532284001 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2858376K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007557] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010156] LSM: Security Framework initializing [ 0.011066] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021388] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026623] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029138] *** VALIDATE tmpfs *** [ 0.030497] *** VALIDATE proc *** [ 0.032256] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041424] debug: unmapping init [mem 0xffffffffbb259000-0xffffffffbb260fff] [ 0.043876] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052317] rcu: Hierarchical SRCU implementation. [ 0.054588] smp: Bringing up secondary CPUs ... [ 0.055571] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.059278] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144879] node 0 deferred pages initialised in 80ms [ 0.149008] devtmpfs: initialized [ 0.150306] x86/mm: Memory block size: 128MB [ 0.154977] gcov: version magic: 0x41383552 [ 0.157356] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164471] pinctrl core: initialized pinctrl subsystem [ 0.167269] [ 0.167835] ************************************************************* [ 0.170014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173017] ** ** [ 0.175013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178016] ** ** [ 0.181027] ** This means that this kernel is built to expose internal ** [ 0.183012] ** IOMMU data structures, which may compromise security on ** [ 0.186017] ** your system. ** [ 0.188013] ** ** [ 0.190015] ** If you see this message and you are not debugging the ** [ 0.193014] ** kernel, report this immediately to your vendor! ** [ 0.195013] ** ** [ 0.197026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200014] ************************************************************* [ 0.204129] NET: Registered protocol family 16 [ 0.206633] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209218] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217097] cpuidle: using governor menu [ 0.218753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.220624] PCI: Using configuration type 1 for base access [ 0.223161] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233340] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.234017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236033] cryptd: max_cpu_qlen set to 1000 [ 0.238253] ACPI: Added _OSI(Module Device) [ 0.239013] ACPI: Added _OSI(Processor Device) [ 0.241013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246698] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254106] ACPI: Interpreter enabled [ 0.255073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.257013] ACPI: Using IOAPIC for interrupt routing [ 0.259161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.263337] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272231] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.275055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.282107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287599] acpiphp: Slot [2] registered [ 0.289098] acpiphp: Slot [5] registered [ 0.291117] acpiphp: Slot [6] registered [ 0.292114] acpiphp: Slot [7] registered [ 0.295158] acpiphp: Slot [8] registered [ 0.296123] acpiphp: Slot [9] registered [ 0.298145] acpiphp: Slot [10] registered [ 0.299116] acpiphp: Slot [3] registered [ 0.300109] acpiphp: Slot [4] registered [ 0.302102] acpiphp: Slot [11] registered [ 0.303111] acpiphp: Slot [12] registered [ 0.305123] acpiphp: Slot [13] registered [ 0.306123] acpiphp: Slot [14] registered [ 0.307141] acpiphp: Slot [15] registered [ 0.309122] acpiphp: Slot [16] registered [ 0.311151] acpiphp: Slot [17] registered [ 0.312091] acpiphp: Slot [18] registered [ 0.314104] acpiphp: Slot [19] registered [ 0.315105] acpiphp: Slot [20] registered [ 0.317144] acpiphp: Slot [21] registered [ 0.318106] acpiphp: Slot [22] registered [ 0.320121] acpiphp: Slot [23] registered [ 0.321110] acpiphp: Slot [24] registered [ 0.323106] acpiphp: Slot [25] registered [ 0.326131] acpiphp: Slot [26] registered [ 0.328161] acpiphp: Slot [27] registered [ 0.329111] acpiphp: Slot [28] registered [ 0.331115] acpiphp: Slot [29] registered [ 0.332114] acpiphp: Slot [30] registered [ 0.334109] acpiphp: Slot [31] registered [ 0.335096] PCI host bridge to bus 0000:00 [ 0.337021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.339021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.354109] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356060] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.365833] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371013] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.376015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383849] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390428] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.412942] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.418432] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467559] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512802] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.521020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.546020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555907] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.563020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.588020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.599770] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.639021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.650553] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.662020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.674022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.706055] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708461] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.712123] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.714399] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.717303] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.723122] iommu: Default domain type: Passthrough [ 0.725514] SCSI subsystem initialized [ 0.727177] ACPI: bus type USB registered [ 0.728132] usbcore: registered new interface driver usbfs [ 0.730110] usbcore: registered new interface driver hub [ 0.732128] usbcore: registered new device driver usb [ 0.734158] pps_core: LinuxPPS API ver. 1 registered [ 0.736013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.739075] PTP clock support registered [ 0.741115] EDAC MC: Ver: 3.0.0 [ 0.744134] PCI: Using ACPI for IRQ routing [ 0.745779] NetLabel: Initializing [ 0.747016] NetLabel: domain hash size = 128 [ 0.749013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751099] NetLabel: unlabeled traffic allowed by default [ 0.753315] vgaarb: loaded [ 0.755295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.757013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.769000] clocksource: Switched to clocksource kvm-clock [ 0.884981] VFS: Disk quotas dquot_6.6.0 [ 0.886530] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.888994] *** VALIDATE ramfs *** [ 0.890604] *** VALIDATE hugetlbfs *** [ 0.892828] pnp: PnP ACPI init [ 0.895304] pnp: PnP ACPI: found 6 devices [ 0.912631] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.915110] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.916504] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917731] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.919838] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.921906] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.924819] NET: Registered protocol family 2 [ 0.926827] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.930079] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932574] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.936896] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.939573] TCP: Hash tables configured (established 65536 bind 65536) [ 0.942068] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944415] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946447] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948600] NET: Registered protocol family 1 [ 0.950879] RPC: Registered named UNIX socket transport module. [ 0.953320] RPC: Registered udp transport module. [ 0.954996] RPC: Registered tcp transport module. [ 0.957181] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.960045] NET: Registered protocol family 44 [ 0.961970] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964168] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966408] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.968867] PCI: CLS 0 bytes, default 64 [ 0.970599] Unpacking initramfs... [ 2.404832] debug: unmapping init [mem 0xffff8ce1fcc54000-0xffff8ce1fffbffff] [ 2.410750] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.413112] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.417311] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.897618] Initialise system trusted keyrings [ 2.899349] Key type blacklist registered [ 2.901349] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.911217] zbud: loaded [ 2.914423] *** VALIDATE nfs *** [ 2.915787] *** VALIDATE nfs4 *** [ 2.917720] pstore: using deflate compression [ 2.921618] Platform Keyring initialized [ 3.015175] NET: Registered protocol family 38 [ 3.017173] Key type asymmetric registered [ 3.018842] Asymmetric key parser 'x509' registered [ 3.021219] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.024467] io scheduler mq-deadline registered [ 3.025941] io scheduler kyber registered [ 3.027204] io scheduler bfq registered [ 3.028838] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.031771] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.033991] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.036592] ACPI: Power Button [PWRF] [ 3.041780] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.047725] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.057654] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.062531] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.073751] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.100249] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.129143] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.134164] Non-volatile memory driver v1.3 [ 3.136033] Linux agpgart interface v0.103 [ 3.170561] virtio_blk virtio1: [vda] 147160 512-byte logical blocks (75.3 MB/71.9 MiB) [ 3.173922] vda: detected capacity change from 0 to 75345920 [ 3.191971] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.195399] vdb: detected capacity change from 0 to 1073741824 [ 3.211053] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214328] vdc: detected capacity change from 0 to 2621440000 [ 3.229786] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.233111] vdd: detected capacity change from 0 to 2621440000 [ 3.246436] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249540] vde: detected capacity change from 0 to 4294967296 [ 3.263613] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.265876] vdf: detected capacity change from 0 to 4294967296 [ 3.271922] libphy: Fixed MDIO Bus: probed [ 3.283787] usbcore: registered new interface driver usbserial_generic [ 3.286479] usbserial: USB Serial support registered for generic [ 3.288912] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.293132] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.295153] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.297588] mousedev: PS/2 mouse device common for all mice [ 3.300488] rtc_cmos 00:05: RTC can wake from S4 [ 3.303185] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.306933] rtc_cmos 00:05: registered as rtc0 [ 3.308770] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.310996] intel_pstate: CPU model not supported [ 3.313738] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317444] hid: raw HID events driver (C) Jiri Kosina [ 3.320327] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.320520] usbcore: registered new interface driver usbhid [ 3.325492] usbhid: USB HID core driver [ 3.327225] drop_monitor: Initializing network drop monitor service [ 3.329231] Initializing XFRM netlink socket [ 3.331930] NET: Registered protocol family 10 [ 3.334599] Segment Routing with IPv6 [ 3.336069] NET: Registered protocol family 17 [ 3.338046] mpls_gso: MPLS GSO support [ 3.345572] RAS: Correctable Errors collector initialized. [ 3.347078] AVX version of gcm_enc/dec engaged. [ 3.348764] AES CTR mode by8 optimization enabled [ 3.421996] sched_clock: Marking stable (3421929176, 0)->(4317320266, -895391090) [ 3.427416] registered taskstats version 1 [ 3.429450] Loading compiled-in X.509 certificates [ 3.431461] zswap: loaded using pool lzo/zbud [ 3.455805] Key type big_key registered [ 3.467911] Key type encrypted registered [ 3.469637] ima: No TPM chip found, activating TPM-bypass! [ 3.472300] ima: Allocated hash algorithm: sha1 [ 3.474226] ima: No architecture policies found [ 3.476159] evm: Initialising EVM extended attributes: [ 3.478406] evm: security.selinux [ 3.479716] evm: security.ima [ 3.480848] evm: security.capability [ 3.482235] evm: HMAC attrs: 0x1 [ 3.484982] rtc_cmos 00:05: setting system clock to 2026-09-11 14:02:49 UTC (1789135369) [ 3.491432] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.494466] debug: unmapping init [mem 0xffffffffbaf82000-0xffffffffbb258fff] [ 3.504083] Write protecting the kernel read-only data: 28672k [ 3.507776] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 3.510879] debug: unmapping init [mem 0xffffffffb9f14000-0xffffffffb9ffffff] [ 3.547376] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.553496] systemd[1]: Detected virtualization kvm. [ 3.554889] systemd[1]: Detected architecture x86-64. [ 3.556166] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.582817] systemd[1]: No hostname configured. [ 3.584682] systemd[1]: Set hostname to . [ 3.586820] random: systemd: uninitialized urandom read (16 bytes read) [ 3.589344] systemd[1]: Initializing machine ID from random generator. [ 3.638713] random: ln: uninitialized urandom read (6 bytes read) [ 3.717429] random: systemd: uninitialized urandom read (16 bytes read) [ 3.720322] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.726905] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.738709] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.337799] device-mapper: uevent: version 1.0.3 [ 4.339862] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.986776] random: fast init done [ 5.001421] virtio_net virtio0 ens2: renamed from eth0 [ 5.053426] scsi host0: ata_piix [ 5.072507] scsi host1: ata_piix [ 5.074158] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.076699] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.652264] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.987703] random: crng init done [ 9.989139] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.285418] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.377849] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.638035] SELinux: Disabled at runtime. [ 11.703318] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.712363] systemd[1]: Detected virtualization kvm. [ 11.714273] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.215753] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.218828] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.224218] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.227700] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.231185] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.239883] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.248185] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispa[ 12.330593] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.635824] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.953046] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.966421] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.087785] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.096411] EDAC sbridge: Ver: 1.1.2 [ 14.674662] Key type dns_resolver registered [ 14.994579] NFS: Registering the id_resolver key type [ 14.997588] Key type id_resolver registered [ 15.000076] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-server login: [ 48.766030] hrtimer: interrupt took 4944576 ns [ 59.703850] libcfs: loading out-of-tree module taints kernel. [ 59.750190] Key type ._llcrypt registered [ 59.752511] Key type .llcrypt registered [ 59.822635] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 78.679707] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 80.169955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.185487] alg: No test for adler32 (adler32-zlib) [ 82.004297] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 82.933806] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 84.695150] Key type lgssc registered [ 87.310113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.244495] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 159.358399] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 177.963310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.073267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.431214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 179.502470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 179.648472] Lustre: lustre-MDT0000: new disk, initializing [ 179.842137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.868751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.546887] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.736506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.850044] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.882982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.886545] Lustre: Skipped 1 previous similar message [ 204.952942] Lustre: lustre-MDT0001: new disk, initializing [ 205.010864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.048144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.058112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.901416] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.665485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.343873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.628671] Lustre: lustre-OST0000: new disk, initializing [ 227.630455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.633698] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.674360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.166534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.188290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.380140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.019482] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.859935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.009402] Lustre: lustre-OST0001: new disk, initializing [ 254.015387] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 254.024983] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.145735] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.426878] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.339578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 264.347765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 264.499743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 278.346608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.939777] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.450728] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.711587] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing yml_node [ 314.972607] Lustre: DEBUG MARKER: Client: 2.17.57.131 [ 317.342455] Lustre: DEBUG MARKER: MDS: 2.17.57.131 [ 320.744277] Lustre: DEBUG MARKER: OSS: 2.17.57.131 [ 323.034593] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 10:08:07 EDT 2026 [ 342.204414] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.001386] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 364.000469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 364.014540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.042473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.027278] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.044523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.057454] Lustre: Skipped 2 previous similar messages [ 365.070370] Lustre: Skipped 3 previous similar messages [ 366.665807] Lustre: server umount lustre-MDT0000 complete [ 371.930429] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135737 with bad export cookie 16589717607708184247 [ 371.951444] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.265117] LustreError: 9193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.269612] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.280640] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 375.290838] LustreError: 9193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 375.343262] Lustre: Skipped 1 previous similar message [ 376.289672] LustreError: 6520:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.318183] LustreError: 6520:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 378.531791] Lustre: server umount lustre-MDT0001 complete [ 384.383894] Lustre: server umount lustre-OST0000 complete [ 391.149985] Lustre: server umount lustre-OST0001 complete [ 399.882959] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 410.684709] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 454.618532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.984575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 455.007615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 455.092226] Lustre: lustre-MDT0000: new disk, initializing [ 455.203218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.243349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 460.458155] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 472.305359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.419821] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 472.471553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 472.474648] Lustre: Skipped 1 previous similar message [ 472.539349] Lustre: lustre-MDT0001: new disk, initializing [ 472.637481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 472.687894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 472.704889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 477.911804] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 488.906518] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 495.947840] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 504.719166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.274182] Lustre: lustre-OST0000: new disk, initializing [ 505.282105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 505.295232] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 505.466378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.081894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 507.103737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.259254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 512.916269] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 524.361837] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 529.561451] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 529.788200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 535.136562] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 535.314839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 538.081237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 538.084044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.097971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 538.108071] Lustre: Skipped 1 previous similar message [ 538.121944] Lustre: Skipped 3 previous similar messages [ 543.211273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 543.225076] Lustre: Skipped 1 previous similar message [ 543.889923] Lustre: server umount lustre-OST0000 complete [ 549.344368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 549.347775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.447584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.449634] Lustre: Skipped 1 previous similar message [ 555.015867] Lustre: server umount lustre-MDT0000 complete [ 558.567957] LustreError: 15095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.588501] LustreError: 15095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 560.046579] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135926 with bad export cookie 16589717607708185780 [ 560.060153] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 560.061689] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.497671] Lustre: server umount lustre-MDT0001 complete [ 565.309698] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:12:10 (1789135930) [ 568.271711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 570.400925] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:12:15 (1789135935) [ 578.788215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.394809] LustreError: 19461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.555708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 584.687656] LustreError: 19462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.109599] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 594.915836] LustreError: 19462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.930971] LustreError: 19462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 595.318870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.067771] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 609.711664] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 616.015028] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 623.574398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 632.328651] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 642.001457] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 655.850181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 655.859452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.859653] LustreError: Skipped 1 previous similar message [ 655.861268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.875798] Lustre: Skipped 2 previous similar messages [ 655.899300] Lustre: Skipped 1 previous similar message [ 657.102087] Lustre: server umount lustre-OST0000 complete [ 662.498207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 667.742654] Lustre: server umount lustre-MDT0000 complete [ 671.201256] LustreError: 19462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.239356] LustreError: 19462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 672.314567] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136038 with bad export cookie 16589717607708186662 [ 672.336730] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.956207] Lustre: server umount lustre-MDT0001 complete [ 689.555996] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 693.434631] Key type lgssc unregistered [ 693.898649] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.928284] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.956368] LNet: Removed LNI 192.168.202.136@tcp [ 696.192183] Key type .llcrypt unregistered [ 696.195545] Key type ._llcrypt unregistered [ 719.718954] Key type ._llcrypt registered [ 719.720849] Key type .llcrypt registered [ 719.841417] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 720.900984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.013227] alg: No test for adler32 (adler32-zlib) [ 722.031922] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 722.220298] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 723.863317] Key type lgssc registered [ 725.537939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 737.947451] loop: module loaded [ 738.058994] loop0: detected capacity change from 0 to 204800000 [ 738.515241] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.529997] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.110949] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.039500] loop0: detected capacity change from 0 to 204800000 [ 749.306947] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.320215] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.342778] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 754.573197] loop0: detected capacity change from 0 to 204800000 [ 760.840621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 760.865412] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.241395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 762.291795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 762.353287] Lustre: lustre-MDT0000: new disk, initializing [ 762.424162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.463542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 767.910376] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 777.719483] loop1: detected capacity change from 0 to 204800000 [ 784.232248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 784.489537] Lustre: lustre-OST07c6: new disk, initializing [ 784.493270] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 784.500807] Lustre: Skipped 1 previous similar message [ 784.506192] Lustre: 26190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 784.576082] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 791.406704] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 791.654860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 791.670074] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 791.719069] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 796.233873] Lustre: 25176:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 796.249899] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 796.270529] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 796.282767] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 796.290174] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 796.300898] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 811.261306] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 811.281940] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.294246] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.300798] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 811.311200] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 811.327303] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.895509] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 816.909882] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 816.925973] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 816.931358] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.940214] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 816.946417] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.952080] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 816.960247] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.969264] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 816.975823] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 816.983359] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 816.991420] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.248755] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 820.264907] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 820.274982] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 820.283847] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.291980] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 820.303108] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.315597] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 820.326478] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.337429] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 820.347300] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.360068] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 820.372037] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.499433] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 825.519581] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 825.527054] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 825.535462] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.542785] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 825.551564] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.573610] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 825.583029] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.593969] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 825.603309] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 825.612669] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 825.622284] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 833.576936] Lustre: 26855:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 833.594143] Lustre: 26855:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages [ 833.614531] Lustre: 26855:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 833.624234] Lustre: 26855:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 833.632456] Lustre: 26855:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 833.638948] Lustre: 26855:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 833.643835] Lustre: 26855:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 833.651411] Lustre: 26855:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 833.656589] Lustre: 26855:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 833.662527] Lustre: 26855:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 833.667518] Lustre: 26855:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 833.673632] Lustre: 26855:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 850.607910] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 850.626902] Lustre: 25177:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages [ 850.631213] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 850.647308] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 850.658952] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 850.670271] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 850.679765] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 850.698606] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 850.709962] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 850.722545] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 850.734210] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 850.751670] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 888.808225] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.813961] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.843250] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 892.566714] Lustre: server umount lustre-OST07c6 complete [ 906.927333] Lustre: server umount lustre-MDT0000 complete [ 925.147209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 928.087681] Key type lgssc unregistered [ 928.347111] LNet: 28752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.361253] LNetError: 28752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.405928] LNet: Removed LNI 192.168.202.136@tcp [ 930.540835] Key type .llcrypt unregistered [ 930.543440] Key type ._llcrypt unregistered [ 953.030909] Key type ._llcrypt registered [ 953.033115] Key type .llcrypt registered [ 953.140859] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 954.833973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.908302] alg: No test for adler32 (adler32-zlib) [ 956.174831] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 956.480816] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 958.167304] Key type lgssc registered [ 959.645281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 968.150534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 968.171472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.686093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.857757] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 979.946518] LustreError: 30021:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.975715] LustreError: 30021:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 985.058968] LustreError: 30022:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.814311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.146595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.617263] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1003.331243] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.598151] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.390206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.736826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.396578] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1034.774948] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1039.289741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1058.786781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.795390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.805930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1059.811325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1059.813713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.815203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1064.946747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.958395] Lustre: Skipped 2 previous similar messages [ 1070.055348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.070787] Lustre: Skipped 1 previous similar message [ 1072.095273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1072.330369] Lustre: server umount lustre-OST0000 complete [ 1080.289755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1080.298386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.301882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.316263] Lustre: Skipped 1 previous similar message [ 1080.331129] Lustre: Skipped 1 previous similar message [ 1083.966312] Lustre: server umount lustre-MDT0000 complete [ 1085.411612] LustreError: 30751:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.420348] LustreError: 30751:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1088.074537] LustreError: 30002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136454 with bad export cookie 11060880901553775854 [ 1088.079451] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.085691] LustreError: 30002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1088.391511] Lustre: server umount lustre-MDT0001 complete [ 1103.088355] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1106.776657] Key type lgssc unregistered [ 1107.168276] LNet: 33769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.189060] LNetError: 33769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.224442] LNet: Removed LNI 192.168.202.136@tcp [ 1108.544259] Key type .llcrypt unregistered [ 1108.547503] Key type ._llcrypt unregistered [ 1128.635947] Key type ._llcrypt registered [ 1128.640161] Key type .llcrypt registered [ 1128.933391] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1131.245664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1131.324148] alg: No test for adler32 (adler32-zlib) [ 1132.627099] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 1132.844390] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1134.658466] Key type lgssc registered [ 1135.901666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1147.772461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1147.824977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.795886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.150902] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1159.175568] LustreError: 35039:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1159.191232] LustreError: 35039:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1164.268839] LustreError: 35040:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.143951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.536565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1172.714889] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1182.075621] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1188.540690] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1195.978277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1196.403359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.794624] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1213.920560] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1220.984547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1237.474603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1237.490784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.524392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.568515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.582787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.588452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.634074] Lustre: Skipped 1 previous similar message [ 1243.765578] Lustre: server umount lustre-OST0000 complete [ 1248.738155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1248.747482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.768317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.808791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.812201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.840362] Lustre: Skipped 1 previous similar message [ 1254.840856] Lustre: server umount lustre-MDT0000 complete [ 1256.932927] LustreError: 35034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.966631] LustreError: 35034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1259.835712] LustreError: 36884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136625 with bad export cookie 4557662697366303111 [ 1259.840130] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.851527] LustreError: 36884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1260.408215] Lustre: server umount lustre-MDT0001 complete [ 1273.252732] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1276.625829] Key type lgssc unregistered [ 1277.070845] LNet: 38605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.076201] LNetError: 38605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.126729] LNet: Removed LNI 192.168.202.136@tcp [ 1278.722070] Key type .llcrypt unregistered [ 1278.723333] Key type ._llcrypt unregistered [ 1307.226619] Key type ._llcrypt registered [ 1307.229664] Key type .llcrypt registered [ 1307.326507] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1308.398805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.589308] alg: No test for adler32 (adler32-zlib) [ 1309.720338] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 1310.090294] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1311.815748] Key type lgssc registered [ 1312.974308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1323.329302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1323.352451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.976995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.508409] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1334.248360] LustreError: 40025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.281121] LustreError: 40025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1339.367785] LustreError: 40024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.481585] LustreError: 40025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.014447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.524627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1349.322449] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1358.464441] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.697972] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1375.188593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1375.710173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.951911] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1393.672751] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1394.152784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1401.824552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.839380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.855386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.384965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1404.385436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.398710] Lustre: Skipped 1 previous similar message [ 1404.444695] Lustre: Skipped 1 previous similar message [ 1407.283039] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1407.293902] Lustre: Skipped 2 previous similar messages [ 1408.115753] Lustre: server umount lustre-MDT0000 complete [ 1409.506463] LustreError: 40025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.561761] LustreError: 40025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1413.153045] LustreError: 40006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136779 with bad export cookie 9169467904290427040 [ 1413.195907] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1414.624822] LustreError: 40019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1414.625527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.662487] LustreError: 40019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1414.707392] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1419.997150] Lustre: server umount lustre-MDT0001 complete [ 1436.219274] Lustre: server umount lustre-OST0000 complete [ 1456.146478] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1459.093623] Key type lgssc unregistered [ 1459.390864] LNet: 43893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.397327] LNetError: 43893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.415982] LNet: Removed LNI 192.168.202.136@tcp [ 1460.214864] Key type .llcrypt unregistered [ 1460.217278] Key type ._llcrypt unregistered [ 1490.078679] Key type ._llcrypt registered [ 1490.082624] Key type .llcrypt registered [ 1490.293698] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1492.381261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1492.420656] alg: No test for adler32 (adler32-zlib) [ 1493.552756] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 1493.916807] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1495.600088] Key type lgssc registered [ 1496.959575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1506.767507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1506.802102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.524654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.547473] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1518.084705] LustreError: 45313:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1518.111540] LustreError: 45313:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1523.170432] LustreError: 45314:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.221211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.847167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1532.043319] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1541.631742] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1550.255440] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.901297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1560.396502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.654697] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1578.885808] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1578.952545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1586.031641] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1588.194804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1588.207583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.241092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1589.218146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1589.273431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1589.285703] Lustre: Skipped 1 previous similar message [ 1592.067266] Lustre: server umount lustre-OST0000 complete [ 1613.793950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1613.813315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.829413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1613.841688] Lustre: Skipped 1 previous similar message [ 1618.368650] Lustre: server umount lustre-MDT0000 complete [ 1620.963318] LustreError: 45310:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.989409] LustreError: 45310:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1622.971401] LustreError: 45294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136988 with bad export cookie 1937577249327481956 [ 1622.990980] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.994515] LustreError: 45294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1637.343191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1637.706948] Lustre: server umount lustre-MDT0001 complete [ 1651.418561] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1655.911146] Key type lgssc unregistered [ 1656.395600] LNet: 49028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.407159] LNetError: 49028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.453860] LNet: Removed LNI 192.168.202.136@tcp [ 1658.376885] Key type .llcrypt unregistered [ 1658.380252] Key type ._llcrypt unregistered [ 1692.095937] Key type ._llcrypt registered [ 1692.098378] Key type .llcrypt registered [ 1692.180388] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1693.781497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.865602] alg: No test for adler32 (adler32-zlib) [ 1694.823676] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 1695.126640] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1696.951554] Key type lgssc registered [ 1697.955176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1708.517795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1708.537978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.840116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.664166] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1719.266635] LustreError: 50452:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.288171] LustreError: 50452:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1724.384149] LustreError: 50451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1724.832141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.330586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1731.565629] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1741.022325] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1748.714193] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1757.318697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.972572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.545536] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1775.667636] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1779.446427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1785.673697] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1790.794039] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1803.623893] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1110h32m47s [ 1805.771077] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1810.401720] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.404554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.436484] Lustre: Skipped 2 previous similar messages [ 1810.452793] Lustre: Skipped 2 previous similar messages [ 1811.488728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.506624] Lustre: Skipped 2 previous similar messages [ 1813.365733] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1813.380709] Lustre: Skipped 1 previous similar message [ 1814.929653] Lustre: server umount lustre-MDT0000 complete [ 1817.597490] LustreError: 50448:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.631034] LustreError: 50448:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1819.757144] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1823.734110] LustreError: 50451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.767949] LustreError: 50451:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1828.677032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.023826] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.388675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.457436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1830.386040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1835.422710] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1839.839279] Lustre: DEBUG MARKER: Wait for df (36777) ... [ 1841.905215] Lustre: DEBUG MARKER: done [ 1847.264297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1847.273215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.285355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.824464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.851098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.406298] Lustre: server umount lustre-OST0000 complete [ 1861.088980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1861.093240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.095481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.099726] Lustre: Skipped 1 previous similar message [ 1861.123016] Lustre: Skipped 4 previous similar messages [ 1872.351183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1872.629450] Lustre: server umount lustre-MDT0000 complete [ 1874.409832] LustreError: 50447:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.430746] LustreError: 50447:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1877.270043] LustreError: 50433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137243 with bad export cookie 2053553874471041317 [ 1877.277829] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.294185] LustreError: 50433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1877.822664] Lustre: server umount lustre-MDT0001 complete [ 1890.848496] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1893.972590] Key type lgssc unregistered [ 1894.360882] LNet: 55383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.374049] LNetError: 55383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.393896] LNet: Removed LNI 192.168.202.136@tcp [ 1895.342669] Key type .llcrypt unregistered [ 1895.344856] Key type ._llcrypt unregistered [ 1942.573518] Key type ._llcrypt registered [ 1942.577252] Key type .llcrypt registered [ 1942.743193] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1944.372953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.437452] alg: No test for adler32 (adler32-zlib) [ 1945.626154] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 1945.945322] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1947.631740] Key type lgssc registered [ 1948.958233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1958.331711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1958.368100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.718156] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1959.764827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1960.010234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.472244] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1976.363359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.529933] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1976.586024] Lustre: 57602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1977.071809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.001908] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1994.022597] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2001.387982] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.564406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.029853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2011.418442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.633634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2019.997769] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2030.035324] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.979064] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1110h36m41s [ 2039.930627] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2048.345794] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2052.074593] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2052.167437] LustreError: 58868:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2072.419846] Lustre: lustre-MDT0000: Client 100f824a-254a-463f-8ba1-03c619483aba (at 192.168.202.36@tcp) reconnecting [ 2092.208082] LustreError: 58868:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2096.022638] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2099.986955] Lustre: DEBUG MARKER: done [ 2111.974650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.977263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2112.009443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2112.024767] Lustre: Skipped 1 previous similar message [ 2113.678318] Lustre: server umount lustre-OST0000 complete [ 2120.672175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2120.677340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.687618] Lustre: Skipped 1 previous similar message [ 2120.693989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.762212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.773383] Lustre: Skipped 2 previous similar messages [ 2123.810967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.965707] Lustre: server umount lustre-MDT0000 complete [ 2129.265113] LustreError: 57595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137495 with bad export cookie 5945736223601171179 [ 2129.296968] LustreError: 57595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2129.299975] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.999096] Lustre: server umount lustre-MDT0001 complete [ 2144.442488] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2148.026887] Key type lgssc unregistered [ 2148.342614] LNet: 61938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.356417] LNetError: 61938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.415513] LNet: Removed LNI 192.168.202.136@tcp [ 2150.877889] Key type .llcrypt unregistered [ 2150.884525] Key type ._llcrypt unregistered [ 2205.856940] Key type ._llcrypt registered [ 2205.858941] Key type .llcrypt registered [ 2205.986185] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2207.172657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.512230] alg: No test for adler32 (adler32-zlib) [ 2208.822222] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 2209.136726] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2210.871635] Key type lgssc registered [ 2212.521893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.202236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.227264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.618805] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2217.659690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2217.915663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.335717] Lustre: Failing over lustre-MDT0000 [ 2219.782841] Lustre: server umount lustre-MDT0000 complete [ 2229.003732] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:39:53 (1789137593) [ 2237.409777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.924113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.232866] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2254.564981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.729280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2254.792609] Lustre: 64740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2255.232495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.902459] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2271.843732] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2279.087505] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.193138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.562941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2288.966297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.711444] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2297.083963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2306.493438] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2312.859746] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2321.888329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2321.902185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.928768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.927219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.937601] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.956743] Lustre: Skipped 1 previous similar message [ 2327.199937] Lustre: server umount lustre-MDT0000 complete [ 2331.310745] LustreError: 64731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137697 with bad export cookie 10144066354194828791 [ 2331.326815] LustreError: 64731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.329481] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2332.102225] Lustre: server umount lustre-MDT0001 complete [ 2336.457412] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2342.378159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2349.547763] Lustre: 63660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137699/real 1789137699] req@0000000096d102a4 x1876046721887744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2349.579654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.671413] Lustre: 63660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137702/real 1789137702] req@00000000f97635e3 x1876046721887872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2354.661216] Lustre: 63663:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137704/real 1789137704] req@000000004b841783 x1876046721888000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2355.617785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2356.772391] LustreError: 63659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001a09234a x1876046721888768/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2357.201739] LustreError: 66587:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.222825] LustreError: 66587:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2357.334758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.477208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2358.304528] Lustre: 63662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137707/real 1789137707] req@00000000e5cf9046 x1876046721888256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2358.376462] LustreError: 66588:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.459445] LustreError: 68513:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.491570] LustreError: 68513:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2362.468968] Lustre: 63661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137712/real 1789137712] req@00000000c116c774 x1876046721888384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2363.341964] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2364.905367] LustreError: 66587:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.926539] LustreError: 66587:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2370.036286] LustreError: 66588:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.061746] LustreError: 66588:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2372.866616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.152516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.834282] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2378.226440] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2379.317124] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2387.533861] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.589875] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2403.817874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2403.827198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2403.866223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.870225] Lustre: Skipped 2 previous similar messages [ 2408.933153] Lustre: server umount lustre-MDT0000 complete [ 2409.953806] LustreError: 68514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.972260] LustreError: 68514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2413.888857] LustreError: 66605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137779 with bad export cookie 10144066354194831066 [ 2413.904621] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.917549] LustreError: 66605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2414.258986] Lustre: server umount lustre-MDT0001 complete [ 2421.741729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2431.455177] Lustre: 63662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137781/real 1789137781] req@00000000adce38f2 x1876046721918464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789137797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2431.488360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.501214] Lustre: Skipped 2 previous similar messages [ 2432.927698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.689373] LustreError: 63659:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002d89e99a x1876046721919488/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2439.157812] LustreError: 66588:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.322374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2439.403882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2444.286433] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2454.775548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.325635] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2460.834273] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2469.075112] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.837218] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2486.241027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.248512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.259394] Lustre: Skipped 2 previous similar messages [ 2486.270572] Lustre: Skipped 5 previous similar messages [ 2491.364048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2491.380784] Lustre: Skipped 2 previous similar messages [ 2492.540092] Lustre: server umount lustre-MDT0000 complete [ 2496.480668] LustreError: 66587:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.517797] LustreError: 66587:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 2497.151706] LustreError: 66605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137863 with bad export cookie 10144066354194832445 [ 2497.161212] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.163230] LustreError: 66605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2497.845677] Lustre: server umount lustre-MDT0001 complete [ 2505.783675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2510.320781] Lustre: DEBUG MARKER: files compared the same [ 2522.873562] Lustre: server umount lustre-OST0000 complete [ 2543.023699] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2545.228655] Key type lgssc unregistered [ 2545.578712] LNet: 74479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.585722] LNetError: 74479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.599933] LNet: Removed LNI 192.168.202.136@tcp [ 2546.679479] Key type .llcrypt unregistered [ 2546.681489] Key type ._llcrypt unregistered [ 2578.107548] Key type ._llcrypt registered [ 2578.109510] Key type .llcrypt registered [ 2578.224171] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2579.635141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2579.757188] alg: No test for adler32 (adler32-zlib) [ 2580.829934] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 2581.133753] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2582.864300] Key type lgssc registered [ 2584.255164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2593.292353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2593.309132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.864910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.069787] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2604.001225] LustreError: 75899:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.017504] LustreError: 75899:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2609.121847] LustreError: 75900:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.895322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.183857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.181759] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2624.230876] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2631.665421] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2640.420278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.960106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.742654] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2655.231563] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2664.499274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2676.712959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.718531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.724392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.434296] Lustre: server umount lustre-OST0000 complete [ 2682.849254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2682.854258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.870336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.896230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.896983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.914316] Lustre: Skipped 1 previous similar message [ 2690.019387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.042512] Lustre: Skipped 1 previous similar message [ 2692.144547] Lustre: server umount lustre-MDT0000 complete [ 2694.123868] LustreError: 78391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.140758] LustreError: 78391:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2696.393854] LustreError: 75881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138062 with bad export cookie 13140521879258209646 [ 2696.407514] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.410735] LustreError: 75881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2696.828709] Lustre: server umount lustre-MDT0001 complete [ 2710.553676] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2713.926475] Key type lgssc unregistered [ 2714.196948] LNet: 79457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2714.207312] LNetError: 79457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2714.222840] LNet: Removed LNI 192.168.202.136@tcp [ 2715.327548] Key type .llcrypt unregistered [ 2715.329366] Key type ._llcrypt unregistered [ 2744.928823] Key type ._llcrypt registered [ 2744.931171] Key type .llcrypt registered [ 2745.052629] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2746.685405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2746.782263] alg: No test for adler32 (adler32-zlib) [ 2747.985383] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 2748.263714] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2749.967623] Key type lgssc registered [ 2751.288591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2760.098366] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2760.110381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.647390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2780.772049] LustreError: 80848:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 2807.455250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.536414] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2823.945908] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2832.252056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.409624] Lustre: *** cfs_fail_loc=706, val=0*** [ 2832.767223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.920174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2837.990792] LustreError: 80874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.006660] LustreError: 80874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2838.403944] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2842.109762] LustreError: 80873:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.129485] LustreError: 80873:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2847.207044] LustreError: 82372:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.242771] LustreError: 82372:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2848.475455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.939553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2854.045864] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2862.650515] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2869.505936] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2873.839988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2873.860393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2873.888899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2874.855834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2874.858154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2874.861023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.903467] Lustre: Skipped 1 previous similar message [ 2878.296259] Lustre: server umount lustre-OST0000 complete [ 2885.093489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.096162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2885.099607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.109289] Lustre: Skipped 1 previous similar message [ 2888.897240] Lustre: server umount lustre-MDT0000 complete [ 2890.210178] LustreError: 81939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2890.223527] LustreError: 81939:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2892.839352] LustreError: 80854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138258 with bad export cookie 6875582787288406622 [ 2892.846698] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2892.854661] LustreError: 80854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2893.348166] Lustre: server umount lustre-MDT0001 complete [ 2906.872210] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2910.235960] Key type lgssc unregistered [ 2910.485067] LNet: 84471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.490374] LNetError: 84471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2910.506793] LNet: Removed LNI 192.168.202.136@tcp [ 2911.468254] Key type .llcrypt unregistered [ 2911.470433] Key type ._llcrypt unregistered [ 2943.107669] Key type ._llcrypt registered [ 2943.116880] Key type .llcrypt registered [ 2943.254756] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2944.910337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2944.941987] alg: No test for adler32 (adler32-zlib) [ 2946.090288] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 2946.304460] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2947.983228] Key type lgssc registered [ 2949.529637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.754212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.802237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.373760] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2974.682801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.231792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2980.329849] LustreError: 86578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.351866] LustreError: 86578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2980.915936] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2985.450218] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.475850] LustreError: 86581:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2990.018155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.393634] LustreError: 86582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2990.414238] LustreError: 86582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2990.439618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.681163] LustreError: 86578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.735407] LustreError: 86578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2997.941104] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3000.821748] LustreError: 86582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.840369] LustreError: 86582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3006.498524] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.537770] Lustre: MGS: Client 845f380e-c540-4c99-899c-45d507275d2b (at 0@lo) reconnecting [ 3006.567482] LustreError: 86592:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3006.848344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.980318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3011.667426] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3027.425529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.428233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.439080] LustreError: Skipped 1 previous similar message [ 3027.452989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.549543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3032.557262] Lustre: Skipped 1 previous similar message [ 3033.712668] Lustre: server umount lustre-OST0000 complete [ 3042.789198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3042.793737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.813017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3042.813025] Lustre: Skipped 1 previous similar message [ 3042.851779] Lustre: Skipped 2 previous similar messages [ 3044.378941] Lustre: server umount lustre-MDT0000 complete [ 3047.909263] LustreError: 86582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.915469] LustreError: 86582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3048.343127] Lustre: server umount lustre-MDT0001 complete [ 3052.426791] Lustre: server umount MGS complete [ 3066.255699] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:53:50 (1789138430) [ 3102.054693] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3115.228292] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3156.258789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.354227] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3173.231287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.651413] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3173.739582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3173.831511] Lustre: lustre-MDT0001: new disk, initializing [ 3173.939519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3173.984120] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3174.006293] Lustre: 93770:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3174.016120] LustreError: 93770:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 3179.604671] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3190.886374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.183966] Lustre: lustre-OST0000: new disk, initializing [ 3191.188643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3191.196904] Lustre: 94536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3191.280620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3197.538975] Lustre: 94772:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f9520611 x1876047494684416/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789138579 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3198.483461] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3211.449718] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3217.887754] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3220.639219] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.651187] Lustre: MGS: Client 20c209e7-59e6-403e-a2ab-ea5abafcec14 (at 0@lo) reconnecting [ 3220.688239] Lustre: 93036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3220.711846] LustreError: 93756:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3220.722233] LustreError: 93756:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3220.786291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3220.956380] Lustre: lustre-MDT0000: new disk, initializing [ 3221.173461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.214781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3222.303988] Lustre: 95609:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000027d1c4 x1876047494691840/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789138604 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3227.360580] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3227.543631] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3231.433658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3231.441552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3231.576840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3256.293166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.296505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3256.318704] Lustre: Skipped 1 previous similar message [ 3256.326368] Lustre: Skipped 2 previous similar messages [ 3261.420165] LustreError: 94521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.620669] Lustre: server umount lustre-OST0000 complete [ 3266.532614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.536402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.557607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.583964] Lustre: Skipped 1 previous similar message [ 3266.596728] Lustre: Skipped 1 previous similar message [ 3276.776381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.788191] Lustre: Skipped 3 previous similar messages [ 3280.351470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3280.474770] Lustre: server umount lustre-MDT0000 complete [ 3300.319196] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3300.856437] Lustre: server umount lustre-MDT0001 complete [ 3306.510541] Lustre: server umount MGS complete [ 3323.112867] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:58:07 (1789138687) [ 3346.731684] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3350.354884] Key type lgssc unregistered [ 3350.585766] LNet: 98116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3350.593295] LNetError: 98116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3350.615237] LNet: Removed LNI 192.168.202.136@tcp [ 3351.710443] Key type .llcrypt unregistered [ 3351.711751] Key type ._llcrypt unregistered [ 3375.783568] Key type ._llcrypt registered [ 3375.785942] Key type .llcrypt registered [ 3375.975307] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3377.250838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3377.302377] alg: No test for adler32 (adler32-zlib) [ 3379.094321] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 3379.473845] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3381.298572] Key type lgssc registered [ 3382.439606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3393.095478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3393.118973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.953345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.640808] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3404.258206] LustreError: 99337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.294369] LustreError: 99337:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3409.383325] LustreError: 99338:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.531657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.805336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3416.907452] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3428.371329] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3435.164924] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3443.750320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.090249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3451.272284] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3460.617093] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3464.578603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3474.924672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3474.930569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.940468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3474.947136] Lustre: Skipped 1 previous similar message [ 3474.963036] Lustre: Skipped 1 previous similar message [ 3480.033350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3480.554136] Lustre: server umount lustre-OST0000 complete [ 3485.155176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.158843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3485.186225] Lustre: Skipped 1 previous similar message [ 3485.208841] Lustre: Skipped 2 previous similar messages [ 3490.275171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.303105] Lustre: Skipped 1 previous similar message [ 3491.025206] Lustre: server umount lustre-MDT0000 complete [ 3495.392987] LustreError: 99337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.424389] LustreError: 99337:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3496.056526] LustreError: 99318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138862 with bad export cookie 11819876360267541012 [ 3496.063384] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3496.072919] LustreError: 99318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3496.453466] Lustre: server umount lustre-MDT0001 complete [ 3512.167670] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3515.188161] Key type lgssc unregistered [ 3515.514869] LNet: 102910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3515.519343] LNetError: 102910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3516.588612] LNet: Removed LNI 192.168.202.136@tcp [ 3517.732285] Key type .llcrypt unregistered [ 3517.734621] Key type ._llcrypt unregistered [ 3538.813766] Key type ._llcrypt registered [ 3538.819147] Key type .llcrypt registered [ 3538.983803] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3540.391375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3540.451185] alg: No test for adler32 (adler32-zlib) [ 3541.441661] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 3541.684219] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3543.415403] Key type lgssc registered [ 3545.157476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3552.993454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3552.997167] LustreError: 103997:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3553.000713] Lustre: Skipped 1 previous similar message [ 3553.000732] LustreError: 103994:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3553.040714] LustreError: 103997:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3553.042978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.052911] LustreError: 103997:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3553.052939] LustreError: 103997:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3554.985451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.514772] LustreError: 104034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3564.541401] LustreError: 104034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3565.769559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.305146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3571.016439] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3577.440991] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing lsmod [ 3580.307776] LustreError: 105202:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3580.329796] LustreError: 105204:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3580.336887] LustreError: 105202:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3580.340738] LustreError: 105202:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3580.352939] LustreError: 105202:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3580.360661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3580.838041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.031427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.045379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.046411] Lustre: Skipped 1 previous similar message [ 3586.060177] Lustre: Skipped 2 previous similar messages [ 3591.136640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.143623] Lustre: Skipped 2 previous similar messages [ 3592.163156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3592.166754] Lustre: Skipped 1 previous similar message [ 3597.284027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.296137] Lustre: Skipped 2 previous similar messages [ 3598.303249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3598.640514] Lustre: server umount lustre-MDT0000 complete [ 3602.412132] LustreError: 105388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.430574] LustreError: 105388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3602.827885] LustreError: 104013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138968 with bad export cookie 4944569050504247689 [ 3602.841712] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3602.846388] LustreError: 104013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3604.575375] LustreError: 104466:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003a307b9d x1876048118569728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3604.857144] Lustre: server umount lustre-MDT0001 complete [ 3619.244747] Lustre: server umount lustre-OST0000 complete [ 3629.347226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.730343] LustreError: 106272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3629.785709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.524657] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3635.194160] LustreError: 106273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.278235] LustreError: 106272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.603149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.172520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3653.189527] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3664.119901] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3671.289360] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.148734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.616103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.001988] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3698.961026] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3699.074092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3709.413276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3709.417122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3709.418133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3709.440679] Lustre: Skipped 2 previous similar messages [ 3715.048501] Lustre: server umount lustre-OST0000 complete [ 3719.648405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.687160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3719.705023] Lustre: Skipped 4 previous similar messages [ 3725.489410] Lustre: server umount lustre-MDT0000 complete [ 3729.836738] LustreError: 106252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139095 with bad export cookie 4944569050504248389 [ 3729.852435] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3729.860913] LustreError: 106252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3730.246244] Lustre: server umount lustre-MDT0001 complete [ 3744.568403] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3747.614885] Key type lgssc unregistered [ 3747.976561] LNet: 109811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3747.981892] LNetError: 109811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3748.012308] LNet: Removed LNI 192.168.202.136@tcp [ 3749.000959] Key type .llcrypt unregistered [ 3749.002538] Key type ._llcrypt unregistered [ 3779.316442] Key type ._llcrypt registered [ 3779.318806] Key type .llcrypt registered [ 3779.441034] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3780.909622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3781.104633] alg: No test for adler32 (adler32-zlib) [ 3782.493734] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 3783.070582] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3784.927187] Key type lgssc registered [ 3786.330609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3796.148858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3796.198461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.769190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.770827] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3807.211234] LustreError: 111228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.247896] LustreError: 111228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3812.325553] LustreError: 111227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.423978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.774830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3819.024410] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3829.129952] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3835.434687] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3842.926080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.457908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.769883] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3859.407771] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3868.022452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3876.426786] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3881.010443] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3886.567152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.568280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.580752] Lustre: Skipped 1 previous similar message [ 3886.609558] Lustre: Skipped 1 previous similar message [ 3891.593413] Lustre: server umount lustre-OST0000 complete [ 3895.778843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3895.782969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.798217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.800501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.803375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.822873] Lustre: Skipped 1 previous similar message [ 3901.921164] LustreError: 112324:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3902.164745] Lustre: server umount lustre-MDT0000 complete [ 3906.020721] LustreError: 111227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.038935] LustreError: 111227:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3906.153906] LustreError: 111208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139272 with bad export cookie 2082643387539220359 [ 3906.156544] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.166097] LustreError: 111208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3906.559950] Lustre: server umount lustre-MDT0001 complete [ 3920.126084] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3923.444690] Key type lgssc unregistered [ 3923.869962] LNet: 114895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3923.874461] LNetError: 114895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3923.886934] LNet: Removed LNI 192.168.202.136@tcp [ 3924.664748] Key type .llcrypt unregistered [ 3924.667144] Key type ._llcrypt unregistered [ 3947.507798] Key type ._llcrypt registered [ 3947.513244] Key type .llcrypt registered [ 3947.736075] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3950.007915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3950.067351] alg: No test for adler32 (adler32-zlib) [ 3951.471781] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 3951.999611] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3953.767272] Key type lgssc registered [ 3955.522335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3964.174265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3964.204247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.822292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.457396] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3975.146313] LustreError: 116117:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.157914] LustreError: 116117:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3979.782441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.135501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.080038] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3994.241497] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4002.893151] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4011.418403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.693639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.441742] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4027.347319] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4035.278935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4046.824457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4046.828242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.839358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.925577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.925846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.927374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.938559] Lustre: Skipped 1 previous similar message [ 4052.869507] Lustre: server umount lustre-OST0000 complete [ 4061.166093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.168878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.184157] Lustre: Skipped 1 previous similar message [ 4063.717766] Lustre: server umount lustre-MDT0000 complete [ 4066.274037] LustreError: 116116:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.295082] LustreError: 116116:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4067.964484] LustreError: 116098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139433 with bad export cookie 11151781370855531128 [ 4067.972305] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4067.972790] LustreError: 116098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4068.296305] Lustre: server umount lustre-MDT0001 complete [ 4081.316479] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4084.006802] Key type lgssc unregistered [ 4084.288283] LNet: 119677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4084.294033] LNetError: 119677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4084.310863] LNet: Removed LNI 192.168.202.136@tcp [ 4084.946100] Key type .llcrypt unregistered [ 4084.948394] Key type ._llcrypt unregistered [ 4118.301137] Key type ._llcrypt registered [ 4118.302655] Key type .llcrypt registered [ 4118.383908] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4119.362116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4119.418296] alg: No test for adler32 (adler32-zlib) [ 4120.479488] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 4120.795175] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4122.569348] Key type lgssc registered [ 4123.946561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4132.525206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4132.544794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.067255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4139.302575] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4144.634300] LustreError: 121103:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4144.665524] LustreError: 121103:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4148.955053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.349843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4154.290053] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4163.241170] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4169.867743] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4177.868705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.246184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4179.325025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4184.786896] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4193.627534] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4201.510872] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4205.971370] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4205.976068] Lustre: Skipped 2 previous similar messages [ 4210.776069] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4210.838337] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4211.452821] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4211.463653] Lustre: Skipped 2 previous similar messages [ 4220.542765] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4220.549138] Lustre: Skipped 1 previous similar message [ 4225.598304] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4225.613781] Lustre: Skipped 2 previous similar messages [ 4226.431586] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.36@tcp 192.168.0.[2,10]@tcp [ 4235.485959] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.36/24@tcp in log lustre-MDT0000 [ 4235.495282] Lustre: Skipped 2 previous similar messages [ 4245.083516] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.36/24@tcp [ 4245.094389] Lustre: Skipped 1 previous similar message [ 4251.123398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4251.133889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.136561] LustreError: Skipped 2 previous similar messages [ 4251.190391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4252.148713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.157442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4252.164949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4252.194633] LustreError: Skipped 4 previous similar messages [ 4257.244594] Lustre: server umount lustre-OST0000 complete [ 4261.861466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4261.872473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.900494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4261.908894] Lustre: Skipped 1 previous similar message [ 4267.491700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4267.499440] Lustre: Skipped 3 previous similar messages [ 4268.056869] Lustre: server umount lustre-MDT0000 complete [ 4272.302681] LustreError: 121085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139638 with bad export cookie 14635089316592759888 [ 4272.315447] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4272.316195] LustreError: 121085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4272.572419] Lustre: server umount lustre-MDT0001 complete [ 4283.368907] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4286.208938] Key type lgssc unregistered [ 4286.589611] LNet: 126100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4286.604991] LNetError: 126100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4286.623084] LNet: Removed LNI 192.168.202.136@tcp [ 4287.911860] Key type .llcrypt unregistered [ 4287.917473] Key type ._llcrypt unregistered [ 4319.847250] Key type ._llcrypt registered [ 4319.854916] Key type .llcrypt registered [ 4319.987757] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4321.394068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4321.492119] alg: No test for adler32 (adler32-zlib) [ 4322.714980] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 4323.059294] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4324.761263] Key type lgssc registered [ 4325.986430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4335.593948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4335.631379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.378762] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4337.447027] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.36/24@tcp [ 4337.467712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4345.735787] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4347.897104] LustreError: 127664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4347.917275] LustreError: 127664:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4352.993629] LustreError: 127663:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.114799] LustreError: 127664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.207889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4358.560163] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4358.583393] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.36/24@tcp [ 4358.627340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4362.286898] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4369.732804] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4377.013042] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4384.392437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4384.740670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.342834] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4400.459856] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4407.292289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4418.530308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4418.533859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.563944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4419.887749] Lustre: server umount lustre-OST0000 complete [ 4425.198049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4425.211041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.230622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.754419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.762854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.865858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.868374] Lustre: Skipped 1 previous similar message [ 4435.124634] Lustre: server umount lustre-MDT0000 complete [ 4437.991192] LustreError: 127658:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.011447] LustreError: 127658:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4439.397735] LustreError: 127645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139805 with bad export cookie 7655475297536367532 [ 4439.402599] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4439.406768] LustreError: 127645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4440.136934] Lustre: server umount lustre-MDT0001 complete [ 4453.087885] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4456.248580] Key type lgssc unregistered [ 4456.533504] LNet: 131277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4456.545139] LNetError: 131277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4456.576934] LNet: Removed LNI 192.168.202.136@tcp [ 4457.312664] Key type .llcrypt unregistered [ 4457.317384] Key type ._llcrypt unregistered [ 4486.419695] Key type ._llcrypt registered [ 4486.422200] Key type .llcrypt registered [ 4486.550348] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4501.702454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4502.760016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4502.952769] alg: No test for adler32 (adler32-zlib) [ 4504.156880] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 4504.496292] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4506.183168] Key type lgssc registered [ 4507.420912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4547.439260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4547.473786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.862256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4548.905297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4549.020310] Lustre: lustre-MDT0000: new disk, initializing [ 4549.096748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.122229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4554.485680] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4567.740051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.877954] Lustre: 134684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4567.935610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4567.939560] Lustre: Skipped 1 previous similar message [ 4568.002546] Lustre: lustre-MDT0001: new disk, initializing [ 4568.119156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4568.161586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4568.176880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4572.829395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4583.603741] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4590.182346] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4598.102161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4598.420182] Lustre: lustre-OST0000: new disk, initializing [ 4598.427305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4598.438625] Lustre: 136655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4598.534633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.476817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4604.488993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4604.559938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4605.348206] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4615.665532] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4621.727314] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4622.050803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4627.513218] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4627.820029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4630.502031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.503884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4630.527433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.621681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.633037] Lustre: Skipped 2 previous similar messages [ 4640.743681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.758805] Lustre: Skipped 1 previous similar message [ 4644.319123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4644.739548] Lustre: server umount lustre-OST0000 complete [ 4649.955593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4649.960750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.971743] Lustre: Skipped 1 previous similar message [ 4649.976610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4650.990808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4656.097484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4656.108396] Lustre: Skipped 3 previous similar messages [ 4664.287305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4664.734446] Lustre: server umount lustre-MDT0000 complete [ 4669.465342] LustreError: 134676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140035 with bad export cookie 4785793549285755493 [ 4669.472842] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.475291] LustreError: 134676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4670.198572] Lustre: server umount lustre-MDT0001 complete [ 4679.881280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.453647] LustreError: 138875:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.489801] LustreError: 138875:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4680.561543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4685.802642] LustreError: 138874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.896230] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4690.921181] LustreError: 138875:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.000728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.042298] LustreError: 138874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4696.480637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4701.658324] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4710.899986] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4717.182839] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4725.555025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4725.872246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.483481] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4743.330984] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4749.340667] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4750.832315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4757.337287] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4757.644921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4767.969313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.221361] Lustre: lustre-OST0001: new disk, initializing [ 4768.227939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4768.237494] Lustre: 142252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4768.353161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4776.118431] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4776.643401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4776.659448] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4776.804316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4786.138973] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4786.412431] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4791.968334] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4792.243348] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4799.186821] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9097135f1800.ost_server_uuid 50 [ 4801.047656] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9097135f1800.ost_server_uuid in IDLE state after 0 sec [ 4812.769237] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.771056] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.783085] LustreError: Skipped 1 previous similar message [ 4812.785306] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4812.785313] Lustre: Skipped 2 previous similar messages [ 4812.822265] Lustre: Skipped 1 previous similar message [ 4817.947948] LustreError: 140697:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.073829] Lustre: server umount lustre-OST0001 complete [ 4822.497403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4822.500756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4828.129128] LustreError: 142540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4828.152506] LustreError: 142540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4828.544987] Lustre: server umount lustre-OST0000 complete [ 4833.249130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4833.266485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4833.269697] Lustre: Skipped 2 previous similar messages [ 4833.299247] Lustre: Skipped 7 previous similar messages [ 4838.961979] Lustre: server umount lustre-MDT0000 complete [ 4843.003934] LustreError: 138854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140209 with bad export cookie 4785793549285756375 [ 4843.012633] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4843.570334] Lustre: server umount lustre-MDT0001 complete [ 4866.174312] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4869.124615] Key type lgssc unregistered [ 4869.449131] LNet: 145263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4869.462671] LNetError: 145263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4870.503302] LNet: Removed LNI 192.168.202.136@tcp [ 4871.425353] Key type .llcrypt unregistered [ 4871.427444] Key type ._llcrypt unregistered [ 4918.811993] Key type ._llcrypt registered [ 4918.814216] Key type .llcrypt registered [ 4918.928741] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4936.246300] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4937.445537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.771327] alg: No test for adler32 (adler32-zlib) [ 4938.910576] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 4939.172672] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4940.839582] Key type lgssc registered [ 4942.042777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4979.679597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4979.701486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.970739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4981.016315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4981.122111] Lustre: lustre-MDT0000: new disk, initializing [ 4981.261625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.296936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4987.004177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4999.416396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.668836] Lustre: 149423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4999.828597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4999.839457] Lustre: Skipped 1 previous similar message [ 5000.014869] Lustre: lustre-MDT0001: new disk, initializing [ 5000.195678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5000.241017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5000.251104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.293125] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5018.616549] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5026.877844] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5035.950581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5036.622842] Lustre: lustre-OST0000: new disk, initializing [ 5036.628467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5036.651745] Lustre: 151394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5036.759194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5042.288423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5042.305293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5042.454235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5047.187610] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5060.417975] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5066.694607] Lustre: Failing over lustre-OST0000 [ 5066.980547] Lustre: server umount lustre-OST0000 complete [ 5083.103746] Lustre: 147649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140433/real 1789140433] req@000000001d66f497 x1876049584493184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789140449 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5083.139668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5084.127818] Lustre: 147646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140434/real 1789140434] req@00000000ec32f10a x1876049584493952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140450 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5084.177123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.098874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5089.248627] Lustre: 147647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140439/real 1789140439] req@0000000030c00679 x1876049584495360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5089.265031] Lustre: 147647:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5089.885246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.917484] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5093.953282] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5094.943199] Lustre: 147649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140444/real 1789140444] req@000000008b1cb8b9 x1876049584496640/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5094.990108] Lustre: 147649:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5095.103575] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5095.107537] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5098.252411] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5099.103149] Lustre: 147648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140449/real 1789140449] req@0000000024e7b2ca x1876049584497664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789140465 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5099.124264] Lustre: 147648:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5103.225132] Lustre: Failing over lustre-MDT0000 [ 5103.591397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.809284] Lustre: server umount lustre-MDT0000 complete [ 5108.707861] LustreError: 150795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.734442] LustreError: 150795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 5110.063608] LustreError: 152066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.828943] LustreError: 149874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.852501] LustreError: 149874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5118.949891] LustreError: 149435:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.989498] LustreError: 149435:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5124.065186] Lustre: 147649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789140474/real 1789140474] req@00000000247825bd x1876049584510976/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789140490 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5124.073888] LustreError: 150795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.115497] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5124.140859] LustreError: 150795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5125.179283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.819797] LustreError: 150795:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.844629] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2b2b9b13cdcd2dcd [ 5134.849432] LustreError: 150795:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 5134.856874] Lustre: MGC192.168.202.136@tcp: Connection restored to 0@lo (at 0@lo) [ 5134.878207] Lustre: Skipped 1 previous similar message [ 5135.104975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.166098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5135.668341] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5139.906320] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5140.472635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5140.499108] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5140.549343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5150.699983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.716764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.723798] Lustre: Skipped 3 previous similar messages [ 5150.733075] Lustre: Skipped 1 previous similar message [ 5154.177516] Lustre: server umount lustre-OST0000 complete [ 5160.929119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5160.931087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.953374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5160.981175] Lustre: Skipped 1 previous similar message [ 5161.006723] Lustre: Skipped 1 previous similar message [ 5165.069930] Lustre: server umount lustre-MDT0000 complete [ 5166.052261] LustreError: 149430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.073591] LustreError: 149430:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5169.305313] LustreError: 149416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140535 with bad export cookie 3110750476988919245 [ 5169.311797] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5169.313959] LustreError: 149416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5169.908253] Lustre: server umount lustre-MDT0001 complete [ 5182.292881] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5185.803643] Key type lgssc unregistered [ 5186.275933] LNet: 155060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5186.296410] LNetError: 155060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5186.317986] LNet: Removed LNI 192.168.202.136@tcp [ 5187.443937] Key type .llcrypt unregistered [ 5187.446558] Key type ._llcrypt unregistered [ 5232.191690] Key type ._llcrypt registered [ 5232.195213] Key type .llcrypt registered [ 5232.413279] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5234.024317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5234.236562] alg: No test for adler32 (adler32-zlib) [ 5235.441849] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 5235.933072] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5237.711181] Key type lgssc registered [ 5239.238709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5249.051249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5249.088064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5250.865713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.257446] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5260.260266] LustreError: 156585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.290820] LustreError: 156585:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5265.381039] LustreError: 156586:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5269.698587] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5270.496737] LustreError: 156585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.602234] LustreError: 156614:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5274.622989] LustreError: 156614:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5274.633525] LustreError: 156614:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 5275.038869] Lustre: server umount lustre-MDT0000 complete [ 5294.414057] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5311.080958] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5357.379705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.735658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5357.789178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5357.868843] Lustre: lustre-MDT0000: new disk, initializing [ 5358.011084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.034248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5363.507979] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5375.524244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.717887] Lustre: 159940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5375.832159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5375.842190] Lustre: Skipped 1 previous similar message [ 5375.972112] Lustre: lustre-MDT0001: new disk, initializing [ 5376.065695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5376.112581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5376.129873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5381.671571] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5392.254317] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5400.006413] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5409.289258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5409.814540] Lustre: lustre-OST0000: new disk, initializing [ 5409.828199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5409.838414] Lustre: 161910:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5409.988856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.214261] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5416.547560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5416.567483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5416.740224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5427.926677] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.482075] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5436.941421] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:33:21 (1789140801) [ 5703.648700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5703.650676] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.650896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.687047] Lustre: Skipped 2 previous similar messages [ 5708.768402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5708.772691] Lustre: Skipped 4 previous similar messages [ 5709.212507] Lustre: server umount lustre-MDT0000 complete [ 5713.045558] LustreError: 161909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141079 with bad export cookie 14663132671971515043 [ 5713.053192] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.059526] LustreError: 161909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5713.889114] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.889177] LustreError: 159948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.908409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5719.008961] LustreError: 159949:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.010178] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5719.708458] Lustre: server umount lustre-MDT0001 complete [ 5730.544084] Lustre: server umount lustre-OST0000 complete [ 5742.206604] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5754.137634] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5798.996733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.373856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5799.402747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5799.527108] Lustre: lustre-MDT0000: new disk, initializing [ 5799.622134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.655370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5805.477536] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5819.144610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.351902] Lustre: 166575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5819.413975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5819.416768] Lustre: Skipped 1 previous similar message [ 5819.488969] Lustre: lustre-MDT0001: new disk, initializing [ 5819.603555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5819.639949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5819.654397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5826.779553] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5837.095895] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5844.262822] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5852.659341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5853.135367] Lustre: lustre-OST0000: new disk, initializing [ 5853.143587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5853.148556] Lustre: 168546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5853.252877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.533577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5860.557377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5860.702310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5861.743995] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5874.290871] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5880.225580] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5880.670736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5887.116622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5887.501820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5891.552457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5891.557922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.569593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5891.607393] Lustre: Skipped 1 previous similar message [ 5901.801523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5901.811432] Lustre: Skipped 4 previous similar messages [ 5904.352231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.726264] Lustre: server umount lustre-OST0000 complete [ 5912.032842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5912.041268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.058648] Lustre: Skipped 1 previous similar message [ 5915.951167] Lustre: server umount lustre-MDT0000 complete [ 5917.170437] LustreError: 166588:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.198475] LustreError: 166588:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5920.093917] LustreError: 166569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141286 with bad export cookie 14663132671971673873 [ 5920.099781] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5920.114779] LustreError: 166569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5920.545550] Lustre: server umount lustre-MDT0001 complete [ 5930.536461] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:41:35 (1789141295) [ 5952.278658] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5961.782301] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6005.728983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.080962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6006.117996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6006.209294] Lustre: lustre-MDT0000: new disk, initializing [ 6006.299693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.336941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6012.906888] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6025.173089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.395168] Lustre: 173303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6025.776636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6031.935072] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6044.035587] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6051.309805] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6059.831931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6060.421365] Lustre: lustre-OST0000: new disk, initializing [ 6060.427325] Lustre: Skipped 1 previous similar message [ 6060.434193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6060.438067] Lustre: Skipped 2 previous similar messages [ 6060.443389] Lustre: 175284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6065.776926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6065.798940] Lustre: Skipped 1 previous similar message [ 6065.821039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6065.980668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6069.517375] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6081.413329] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6091.745776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.756593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.791983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6091.798644] Lustre: Skipped 3 previous similar messages [ 6096.125073] Lustre: server umount lustre-OST0000 complete [ 6101.984932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.006634] Lustre: Skipped 1 previous similar message [ 6106.958302] Lustre: server umount lustre-MDT0000 complete [ 6107.108312] LustreError: 173316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.169448] LustreError: 173316:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6111.052067] LustreError: 173295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141477 with bad export cookie 14663132671971674755 [ 6111.060987] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6111.063806] LustreError: 173295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6111.706770] Lustre: server umount lustre-MDT0001 complete [ 6121.042742] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:44:45 (1789141485) [ 6141.552242] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 6152.680836] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6197.212346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.484791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6197.521633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6197.601963] Lustre: lustre-MDT0000: new disk, initializing [ 6197.709697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.712494] Lustre: Skipped 2 previous similar messages [ 6197.731409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6203.037535] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6214.694820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.874289] Lustre: 179608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6215.140761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6220.795304] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6233.102178] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6240.897256] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6249.468446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6249.874604] Lustre: 181589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6255.213841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6255.331005] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6257.821969] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6269.985129] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6281.187873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.194125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6281.208030] Lustre: Skipped 2 previous similar messages [ 6281.221031] Lustre: Skipped 3 previous similar messages [ 6285.756820] Lustre: server umount lustre-OST0000 complete [ 6296.729967] Lustre: server umount lustre-MDT0000 complete [ 6301.218862] LustreError: 180410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141667 with bad export cookie 14663132671971675840 [ 6301.233788] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6301.248388] LustreError: 180410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6301.800570] Lustre: server umount lustre-MDT0001 complete [ 6315.679019] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6319.076986] Key type lgssc unregistered [ 6319.356834] LNet: 183449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6319.364926] LNetError: 183449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6319.383277] LNet: Removed LNI 192.168.202.136@tcp [ 6320.306738] Key type .llcrypt unregistered [ 6320.309327] Key type ._llcrypt unregistered [ 6352.442726] Key type ._llcrypt registered [ 6352.445565] Key type .llcrypt registered [ 6352.620385] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6354.704154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6354.735132] alg: No test for adler32 (adler32-zlib) [ 6356.106716] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 6356.511499] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6358.191651] Key type lgssc registered [ 6360.034950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6372.061776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6372.111213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.732406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6379.747803] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6383.086414] LustreError: 184870:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.112867] LustreError: 184870:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6388.195730] LustreError: 184871:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6391.188531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6391.916763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6398.721761] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6408.548946] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6415.902725] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6423.957308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.536483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.780934] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6440.262560] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6444.042178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6467.554816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6467.559147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6467.579593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6467.596277] Lustre: Skipped 1 previous similar message [ 6467.620258] Lustre: Skipped 1 previous similar message [ 6471.074080] Lustre: server umount lustre-OST0000 complete [ 6477.794072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.799192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.809607] Lustre: Skipped 1 previous similar message [ 6477.832848] Lustre: Skipped 1 previous similar message [ 6482.282121] Lustre: server umount lustre-MDT0000 complete [ 6482.913234] LustreError: 185599:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.945364] LustreError: 185599:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6486.813985] LustreError: 185606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141852 with bad export cookie 9795360437363010779 [ 6486.819781] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6486.832535] LustreError: 185606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6487.337557] Lustre: server umount lustre-MDT0001 complete [ 6502.386961] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6507.138631] Key type lgssc unregistered [ 6507.633288] LNet: 188444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6507.648978] LNetError: 188444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6507.680142] LNet: Removed LNI 192.168.202.136@tcp [ 6508.844238] Key type .llcrypt unregistered [ 6508.846779] Key type ._llcrypt unregistered [ 6541.635749] Key type ._llcrypt registered [ 6541.637582] Key type .llcrypt registered [ 6541.755425] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6543.160167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6543.215113] alg: No test for adler32 (adler32-zlib) [ 6544.751720] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 6545.219976] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6546.999410] Key type lgssc registered [ 6548.439936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6558.198247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6558.234062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.051541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.343197] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6569.449381] LustreError: 189874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.484295] LustreError: 189874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6574.560619] LustreError: 189875:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6577.209955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6577.767916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6583.144175] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6592.791593] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6599.989911] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6606.871046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.230346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6615.522339] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6625.214231] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6629.839743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6636.382898] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6638.051767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.057624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6638.069808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6638.071567] Lustre: Skipped 1 previous similar message [ 6638.092459] Lustre: Skipped 1 previous similar message [ 6641.456750] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6646.580891] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6646.586813] Lustre: Skipped 2 previous similar messages [ 6650.336096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6650.611399] Lustre: server umount lustre-OST0000 complete [ 6660.651687] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff909702cd3000.ost_server_uuid 50 [ 6663.023149] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6679.024576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.040346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.053041] Lustre: Skipped 1 previous similar message [ 6679.082418] Lustre: Skipped 3 previous similar messages [ 6685.158882] Lustre: server umount lustre-MDT0000 complete [ 6685.163090] LustreError: 190818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.202279] LustreError: 190818:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6689.467622] LustreError: 189855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142055 with bad export cookie 3834434718702570286 [ 6689.489245] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6689.713854] Lustre: server umount lustre-MDT0001 complete [ 6701.424705] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:54:25 (1789142065) [ 6709.259507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.894218] LustreError: 193793:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.013640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.372286] LustreError: 193794:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.006611] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6724.577723] LustreError: 193793:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6724.594936] LustreError: 193793:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6726.551833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.052053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.495840] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6742.570992] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6749.089881] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6757.777520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6758.180260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.744915] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6775.520652] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6779.718767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6786.166597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.524678] Lustre: lustre-OST0001: new disk, initializing [ 6786.528216] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6786.535217] Lustre: 196638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6786.654786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6792.289955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6792.301711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6792.458876] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6794.083860] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6805.892444] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6810.593979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6810.606709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6810.641902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6810.657758] Lustre: Skipped 2 previous similar messages [ 6812.641062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6812.654798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.128601] Lustre: server umount lustre-OST0000 complete [ 6817.071680] LustreError: 195617:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6824.011681] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6824.413161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6829.730355] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6830.024521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6849.004993] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6849.017917] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6849.038403] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6849.049650] Lustre: Skipped 4 previous similar messages [ 6852.545749] Lustre: server umount lustre-OST0001 complete [ 6859.236102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.261483] Lustre: Skipped 2 previous similar messages [ 6864.836664] Lustre: server umount lustre-MDT0000 complete [ 6868.963844] LustreError: 196637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142234 with bad export cookie 3834434718702571357 [ 6868.964309] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.968810] LustreError: 196637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6869.264804] Lustre: server umount lustre-MDT0001 complete [ 6898.899338] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:57:43 (1789142263) [ 6906.180487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.517698] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6906.550348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6906.819093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6912.330177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6922.548034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.619210] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6922.647893] Lustre: 200078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6927.947795] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6936.725138] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6943.075323] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6950.357910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6950.482600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6956.481051] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6958.790483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6965.046625] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6973.467772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.585924] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6973.890979] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6973.895335] Lustre: Skipped 2 previous similar messages [ 6981.350566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6981.899375] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6989.602320] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6994.413138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.425522] LustreError: Skipped 1 previous similar message [ 6994.427401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6994.441310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6994.448927] Lustre: Skipped 5 previous similar messages [ 6998.549805] Lustre: server umount lustre-OST0000 complete [ 7000.369994] LustreError: 202424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.394327] LustreError: 202424:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [ 7017.446910] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7017.465350] LustreError: Skipped 1 previous similar message [ 7020.953877] Lustre: server umount lustre-OST0001 complete [ 7027.683619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.718079] Lustre: Skipped 4 previous similar messages [ 7032.197566] Lustre: server umount lustre-MDT0000 complete [ 7036.777757] LustreError: 200073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142402 with bad export cookie 3834434718702572757 [ 7036.782347] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.787847] LustreError: 200073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7037.123878] Lustre: server umount lustre-MDT0001 complete [ 7066.940959] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:00:31 (1789142431) [ 7087.076501] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 7099.254292] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7143.095994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.302923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7143.326867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7143.386732] Lustre: lustre-MDT0000: new disk, initializing [ 7143.436845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7143.457749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7148.230931] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7160.584441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7160.741812] Lustre: 208195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7160.804062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7160.807568] Lustre: Skipped 1 previous similar message [ 7160.878239] Lustre: lustre-MDT0001: new disk, initializing [ 7161.013465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7161.020345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7166.678203] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7177.535605] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.648539] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7192.461908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7192.806308] Lustre: lustre-OST0000: new disk, initializing [ 7192.811516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7192.819777] Lustre: 210164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7199.323224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7199.345346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7199.472586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7200.110557] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7211.294382] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7217.213139] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7217.390604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7223.402111] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7223.736709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7230.434583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7230.452975] LustreError: Skipped 1 previous similar message [ 7230.458471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.471732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7230.480466] Lustre: Skipped 9 previous similar messages [ 7233.129083] Lustre: server umount lustre-OST0000 complete [ 7243.630156] Lustre: server umount lustre-MDT0000 complete [ 7245.797943] LustreError: 208207:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.809972] LustreError: 208207:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 7247.863254] LustreError: 210163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142613 with bad export cookie 3834434718702574549 [ 7247.881857] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.890375] LustreError: 210163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7248.421612] Lustre: server umount lustre-MDT0001 complete [ 7259.912148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.336574] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7277.698715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.290212] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7292.964485] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7299.236326] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7306.883402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7314.557547] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7323.810519] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.541817] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7331.965155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7337.070288] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7337.273833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7339.495175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.509302] LustreError: Skipped 2 previous similar messages [ 7345.607168] Lustre: server umount lustre-OST0000 complete [ 7353.017622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7359.058171] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7391.112660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7392.770647] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7392.816473] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7402.360338] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7406.122861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7406.147622] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7411.131627] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7413.108942] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7421.413737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7421.428406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7421.444392] Lustre: Skipped 6 previous similar messages [ 7421.469250] Lustre: Skipped 7 previous similar messages [ 7424.876907] Lustre: server umount lustre-OST0000 complete [ 7436.542363] Lustre: server umount lustre-MDT0000 complete [ 7436.786048] LustreError: 214874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.808102] LustreError: 214874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7441.473841] LustreError: 212363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142807 with bad export cookie 3834434718702575424 [ 7441.479361] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7441.497128] LustreError: 212363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7442.047599] Lustre: server umount lustre-MDT0001 complete [ 7451.318451] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:06:56 (1789142816) [ 7459.509835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.385518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.392117] Lustre: Skipped 6 previous similar messages [ 7466.594856] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7477.266792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.218566] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7492.881696] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7499.836474] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7507.713108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.063382] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7523.534960] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7529.223961] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7531.750911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7536.796701] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7537.028816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7543.173780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7543.360975] Lustre: lustre-OST0001: new disk, initializing [ 7543.371596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7543.389468] Lustre: 221755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7550.539915] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7550.660434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7550.679861] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7550.851362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7562.006847] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7568.168365] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7568.579862] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7572.760719] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7573.030429] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7576.545655] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7576.557997] LustreError: Skipped 1 previous similar message [ 7589.343562] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7589.579870] Lustre: server umount lustre-OST0001 complete [ 7597.494229] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7597.732418] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7603.574718] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7603.876341] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7635.361719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7641.083184] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7641.100970] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7641.101289] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7641.130841] Lustre: Skipped 1 previous similar message [ 7643.439326] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7652.429411] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7654.309472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7663.548880] Lustre: server umount lustre-OST0001 complete [ 7684.580903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.591238] Lustre: Skipped 6 previous similar messages [ 7684.601054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7684.607359] Lustre: Skipped 10 previous similar messages [ 7688.697110] Lustre: server umount lustre-OST0000 complete [ 7700.028386] Lustre: server umount lustre-MDT0000 complete [ 7702.497539] LustreError: 218369:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7702.532385] LustreError: 218369:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 40 previous similar messages [ 7704.655594] LustreError: 218350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143070 with bad export cookie 3834434718702576397 [ 7704.668900] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7711.343858] Lustre: server umount lustre-MDT0001 complete [ 7740.790726] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:11:45 (1789143105) [ 7751.406002] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.025502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7752.104442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7760.061514] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7771.727334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7771.950320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7772.004840] Lustre: 227005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7779.002937] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7790.745802] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7798.366352] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7808.229847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7808.580414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7816.941747] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7827.726395] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7839.752586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.880991] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7848.659472] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7857.510120] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7859.783823] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7866.959196] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7867.333067] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7876.509082] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7876.901964] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7885.355213] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff909704c92800.ost_server_uuid 50 [ 7887.802488] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff909704c92800.ost_server_uuid in IDLE state after 0 sec [ 7889.442958] Lustre: Permanently deactivating lustre-OST0001 [ 7889.509836] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7893.531226] Lustre: Permanently reactivating lustre-OST0001 [ 7893.571991] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7893.578850] Lustre: Skipped 2 previous similar messages [ 7903.720758] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7903.739658] LustreError: Skipped 4 previous similar messages [ 7905.232922] Lustre: server umount lustre-OST0001 complete [ 7916.814981] Lustre: server umount lustre-OST0000 complete [ 7928.422649] Lustre: server umount lustre-MDT0000 complete [ 7933.240620] LustreError: 229880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143299 with bad export cookie 3834434718702577678 [ 7933.243782] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7933.262305] LustreError: 229880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7933.883677] Lustre: server umount lustre-MDT0001 complete [ 7965.294790] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:15:29 (1789143329) [ 7974.789710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.387714] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7975.443171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7975.471287] Lustre: Skipped 2 previous similar messages [ 7975.884719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7975.897033] Lustre: Skipped 8 previous similar messages [ 7981.085896] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7992.991445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.168036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7993.204395] Lustre: 233711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7999.335682] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8011.180411] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8018.894167] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8027.110727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.443994] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8027.489990] Lustre: Permanently deactivating lustre-OST0000 [ 8027.535419] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8034.908369] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8035.095787] LustreError: 235880:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 8035.106651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8036.635474] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8047.049925] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8055.578336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.701890] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8062.677589] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8064.233193] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8074.643293] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8076.579630] Lustre: Permanently reactivating lustre-OST0000 [ 8076.605033] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8076.609264] Lustre: Skipped 2 previous similar messages [ 8085.638101] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8085.642955] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8085.669933] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8085.753684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8085.782523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8097.879947] Lustre: server umount lustre-OST0001 complete [ 8117.218640] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8117.590458] Lustre: server umount lustre-OST0000 complete [ 8130.616228] Lustre: server umount lustre-MDT0000 complete [ 8136.914847] LustreError: 233701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143502 with bad export cookie 3834434718702579365 [ 8136.918697] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8136.938372] LustreError: 233701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8137.550203] Lustre: server umount lustre-MDT0001 complete [ 8150.682603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8154.892513] Key type lgssc unregistered [ 8155.329180] LNet: 238601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8155.336910] LNetError: 238601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8155.372315] LNet: Removed LNI 192.168.202.136@tcp [ 8156.791113] Key type .llcrypt unregistered [ 8156.793579] Key type ._llcrypt unregistered [ 8187.581262] Key type ._llcrypt registered [ 8187.582414] Key type .llcrypt registered [ 8187.661454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8189.183732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8189.255130] alg: No test for adler32 (adler32-zlib) [ 8190.458529] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 8190.894647] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8192.575287] Key type lgssc registered [ 8193.997350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.891445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8207.905205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.748110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8215.435817] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8219.116576] LustreError: 240074:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.176275] LustreError: 240074:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8224.238224] LustreError: 240075:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.958189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.123678] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8226.183095] Lustre: 240063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8226.190274] Lustre: Permanently deactivating lustre-MDT0001 [ 8226.209349] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8226.546862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8231.932823] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8234.036407] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8241.252142] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8248.606187] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8256.069254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.662249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.473826] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8271.027330] LustreError: 241920:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8276.256665] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8276.464307] LustreError: 241919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.656136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8284.644869] LustreError: 241921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.679316] LustreError: 241921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8286.154050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8286.672073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8291.845523] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8291.858406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8295.121499] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8305.471501] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8309.295928] Lustre: Permanently reactivating lustre-MDT0001 [ 8309.331332] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8309.340261] Lustre: Skipped 2 previous similar messages [ 8312.464972] Lustre: lustre-MDT0001: Client 66b8b724-6bb9-40aa-b8b3-3943dc995c88 (at 192.168.202.36@tcp) reconnecting [ 8319.670437] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.691628] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8319.704358] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8319.723412] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8320.693665] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8322.768234] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8330.194305] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8330.530862] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8335.878502] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8336.174515] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8339.082542] Lustre: Permanently deactivating lustre-MDT0001 [ 8339.122439] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8339.140944] Lustre: Skipped 2 previous similar messages [ 8349.492833] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8363.489940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.495797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8363.537796] Lustre: Skipped 1 previous similar message [ 8368.616109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8368.629280] Lustre: Skipped 6 previous similar messages [ 8369.480491] Lustre: server umount lustre-MDT0000 complete [ 8373.733047] LustreError: 240075:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8373.767872] LustreError: 240075:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8374.721656] LustreError: 240056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143740 with bad export cookie 15100700437031867535 [ 8374.750470] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8375.476269] Lustre: server umount lustre-MDT0001 complete [ 8381.112525] Lustre: server umount lustre-OST0000 complete [ 8396.294194] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:22:40 (1789143760) [ 8409.145812] Lustre: server umount lustre-OST0001 complete [ 8418.144880] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8429.935816] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8480.228331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.709334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8480.714539] Lustre: Skipped 2 previous similar messages [ 8480.737066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8480.843420] Lustre: lustre-MDT0000: new disk, initializing [ 8480.954082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.984779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8489.610678] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8506.484861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.634346] Lustre: 248389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8506.714631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8506.722372] Lustre: Skipped 1 previous similar message [ 8506.800811] Lustre: lustre-MDT0001: new disk, initializing [ 8506.938804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8506.989784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8507.006796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8513.917154] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8526.109312] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8533.681307] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8543.189433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8543.489690] Lustre: lustre-OST0000: new disk, initializing [ 8543.511144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8543.530642] Lustre: 250361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8543.754625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.096858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8551.115220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8551.283105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8553.239647] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8565.543338] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8568.461957] LustreError: 248394:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8578.158289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8578.429907] Lustre: lustre-OST0001: new disk, initializing [ 8578.434985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8578.444254] Lustre: 251475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8578.464173] LustreError: 248394:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8578.511557] LustreError: 248396:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8578.584811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8584.284562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8584.307474] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8584.448802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8586.082422] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8588.583172] LustreError: 248396:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8596.341868] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8598.348921] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8605.155401] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8605.162425] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8605.189303] Lustre: Skipped 2 previous similar messages [ 8605.192817] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8605.235812] Lustre: Skipped 1 previous similar message [ 8606.933990] Lustre: server umount lustre-OST0001 complete [ 8610.273430] LustreError: 250655:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8610.300576] LustreError: 250655:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8625.647955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8625.660934] Lustre: Skipped 2 previous similar messages [ 8625.666366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8625.684771] Lustre: Skipped 1 previous similar message [ 8630.776132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8631.585269] Lustre: server umount lustre-OST0000 complete [ 8635.877592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.993492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8640.998883] Lustre: Skipped 4 previous similar messages [ 8650.207191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8650.771733] Lustre: server umount lustre-MDT0000 complete [ 8651.233815] LustreError: 249381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.267329] LustreError: 249381:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 8655.955678] LustreError: 248381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144021 with bad export cookie 15100700437031870125 [ 8655.962742] LustreError: 248381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8655.963400] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8662.479184] Lustre: server umount lustre-MDT0001 complete [ 8679.798817] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8684.391264] Key type lgssc unregistered [ 8684.854177] LNet: 253516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8684.865329] LNetError: 253516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8684.904968] LNet: Removed LNI 192.168.202.136@tcp [ 8686.281767] Key type .llcrypt unregistered [ 8686.283677] Key type ._llcrypt unregistered [ 8731.114980] Key type ._llcrypt registered [ 8731.117710] Key type .llcrypt registered [ 8731.237422] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8732.955191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8732.985145] alg: No test for adler32 (adler32-zlib) [ 8734.062726] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 8734.476543] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8736.136595] Key type lgssc registered [ 8737.813454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8746.510118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8746.538664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.972948] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8748.007049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8748.252204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8753.480709] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8762.532860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.692710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8762.743985] Lustre: 255726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8763.035766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8768.516825] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8776.537860] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8781.890734] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8790.582201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8790.973480] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8791.429429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.979902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8799.618995] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8808.680963] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8860.643361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.667577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.248827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8865.263675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.279458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8866.770932] Lustre: server umount lustre-OST0000 complete [ 8871.513833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8887.067416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8887.472601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.539598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8890.600243] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8895.593475] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8904.209130] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8905.960239] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8905.976214] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8947.168322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8947.177133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.185161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8947.195540] Lustre: Skipped 1 previous similar message [ 8947.209537] Lustre: Skipped 2 previous similar messages [ 8950.702754] Lustre: server umount lustre-OST0000 complete [ 8957.411624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8957.413091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8957.427928] Lustre: Skipped 1 previous similar message [ 8957.461844] Lustre: Skipped 1 previous similar message [ 8961.641877] Lustre: server umount lustre-MDT0000 complete [ 8965.750308] LustreError: 255719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144331 with bad export cookie 6726896161189810884 [ 8965.754634] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.761631] LustreError: 255719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8966.083346] Lustre: server umount lustre-MDT0001 complete [ 8977.117201] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8980.113308] Key type lgssc unregistered [ 8980.501538] LNet: 261159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8980.508877] LNetError: 261159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8980.538927] LNet: Removed LNI 192.168.202.136@tcp [ 8981.548770] Key type .llcrypt unregistered [ 8981.551509] Key type ._llcrypt unregistered [ 9006.960506] Key type ._llcrypt registered [ 9006.964823] Key type .llcrypt registered [ 9007.175238] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9008.630875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9008.763181] alg: No test for adler32 (adler32-zlib) [ 9009.879574] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 9010.198608] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9011.943973] Key type lgssc registered [ 9013.133827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9022.131084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9022.156948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.820756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9028.643597] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9033.185166] LustreError: 262578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.195881] LustreError: 262578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9037.932537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.409638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9043.373422] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9052.269472] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9059.056900] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9066.666251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9067.042974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9068.098643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 9073.445570] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9081.027360] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9114.099077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9114.132684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9114.135019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9114.152696] LustreError: Skipped 1 previous similar message [ 9114.177833] Lustre: Skipped 1 previous similar message [ 9117.242848] Lustre: server umount lustre-OST0000 complete [ 9124.322405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9124.344877] Lustre: Skipped 1 previous similar message [ 9124.359929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9127.778644] Lustre: server umount lustre-MDT0000 complete [ 9129.440386] LustreError: 262578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9129.477540] LustreError: 262578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9131.947765] LustreError: 262559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144497 with bad export cookie 10830348707314998327 [ 9131.960656] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9131.962103] LustreError: 262559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9132.516458] Lustre: server umount lustre-MDT0001 complete [ 9146.497123] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9149.755697] Key type lgssc unregistered [ 9150.160717] LNet: 266891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9150.165259] LNetError: 266891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9150.188172] LNet: Removed LNI 192.168.202.136@tcp [ 9151.052911] Key type .llcrypt unregistered [ 9151.059594] Key type ._llcrypt unregistered [ 9171.558846] Key type ._llcrypt registered [ 9171.564254] Key type .llcrypt registered [ 9171.698711] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9173.164483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9173.235733] alg: No test for adler32 (adler32-zlib) [ 9174.393255] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 9174.733711] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9176.463527] Key type lgssc registered [ 9177.836399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9187.115078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9187.143939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.569891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9193.988383] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9198.050640] LustreError: 268113:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.067809] LustreError: 268113:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9203.171710] LustreError: 268112:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9203.993403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.470683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9210.289603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9218.787918] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9225.912516] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9234.001531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9234.463408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9242.738091] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9251.610950] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9258.043079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9273.316294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.331268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9273.367032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9273.400788] Lustre: Skipped 1 previous similar message [ 9275.479314] Lustre: server umount lustre-OST0000 complete [ 9281.511144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9281.524054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9281.562784] Lustre: Skipped 1 previous similar message [ 9281.581077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9283.552563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9283.553327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9283.581238] Lustre: Skipped 1 previous similar message [ 9288.689432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.694692] Lustre: Skipped 1 previous similar message [ 9293.793918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9293.818964] Lustre: Skipped 1 previous similar message [ 9294.303191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9294.634052] Lustre: server umount lustre-MDT0000 complete [ 9298.920984] LustreError: 268112:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.944045] LustreError: 268112:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9299.109884] LustreError: 268093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144665 with bad export cookie 11070198422557513875 [ 9299.138762] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9299.146845] LustreError: 268093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9299.795717] Lustre: server umount lustre-MDT0001 complete [ 9314.058319] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9318.045658] Key type lgssc unregistered [ 9318.407374] LNet: 271819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9318.426677] LNetError: 271819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9318.451898] LNet: Removed LNI 192.168.202.136@tcp [ 9319.714682] Key type .llcrypt unregistered [ 9319.718103] Key type ._llcrypt unregistered [ 9347.876210] Key type ._llcrypt registered [ 9347.878296] Key type .llcrypt registered [ 9347.977867] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9349.450038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9349.481653] alg: No test for adler32 (adler32-zlib) [ 9350.810245] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 9351.112880] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9352.991621] Key type lgssc registered [ 9354.676595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9364.548800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.589542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9366.214411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9372.044211] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9376.747017] LustreError: 273238:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9376.780705] LustreError: 273238:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9381.859852] LustreError: 273239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.862120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.314406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9389.414574] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9398.817871] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9406.539921] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9414.670619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9415.005145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.383898] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9431.857693] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9436.540902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9467.366704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9467.375615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9467.394254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9469.409233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9469.423706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.455282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9471.905984] Lustre: server umount lustre-OST0000 complete [ 9480.673228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9480.674615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9480.697043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.708324] Lustre: Skipped 1 previous similar message [ 9480.726979] Lustre: Skipped 3 previous similar messages [ 9485.795993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9485.811115] Lustre: Skipped 1 previous similar message [ 9490.399138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9490.721494] Lustre: server umount lustre-MDT0000 complete [ 9490.919207] LustreError: 273238:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9490.947088] LustreError: 273238:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9495.020591] LustreError: 273239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9495.054684] LustreError: 273239:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9495.484490] LustreError: 273219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789144861 with bad export cookie 18112586141248494688 [ 9495.490765] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9495.497362] LustreError: 273219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9495.972469] Lustre: server umount lustre-MDT0001 complete [ 9508.658401] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9512.054746] Key type lgssc unregistered [ 9512.556660] LNet: 277554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9512.570721] LNetError: 277554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9512.591627] LNet: Removed LNI 192.168.202.136@tcp [ 9513.455897] Key type .llcrypt unregistered [ 9513.459906] Key type ._llcrypt unregistered [ 9532.843429] Key type ._llcrypt registered [ 9532.845321] Key type .llcrypt registered [ 9533.002832] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9534.634494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9534.701249] alg: No test for adler32 (adler32-zlib) [ 9535.806678] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 9536.010915] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9537.663243] Key type lgssc registered [ 9539.014686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9547.539930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9547.580644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.301440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9554.319353] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9559.524827] LustreError: 278793:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9559.542051] LustreError: 278793:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9563.923644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.373402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9569.639933] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9576.795261] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9582.514838] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.311911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.577081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9594.681399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9598.413707] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9607.153717] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9624.555034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.560855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9624.576516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9624.593548] Lustre: Skipped 1 previous similar message [ 9628.642562] Lustre: server umount lustre-OST0000 complete [ 9634.793612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.814320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9634.815061] Lustre: Skipped 2 previous similar messages [ 9634.836047] Lustre: Skipped 1 previous similar message [ 9639.679273] Lustre: server umount lustre-MDT0000 complete [ 9639.904507] LustreError: 278798:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9639.942564] LustreError: 278798:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9644.004983] LustreError: 278792:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9644.038995] LustreError: 278792:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9644.645482] LustreError: 278759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145010 with bad export cookie 7821767777130921784 [ 9644.655889] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9644.671628] LustreError: 278759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9645.219967] Lustre: server umount lustre-MDT0001 complete [ 9659.327059] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9663.313599] Key type lgssc unregistered [ 9663.720728] LNet: 282498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9663.734945] LNetError: 282498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9663.750560] LNet: Removed LNI 192.168.202.136@tcp [ 9665.025511] Key type .llcrypt unregistered [ 9665.030806] Key type ._llcrypt unregistered [ 9681.032739] Key type ._llcrypt registered [ 9681.035766] Key type .llcrypt registered [ 9681.238605] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9704.700209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9722.991966] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9724.591822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9724.851223] alg: No test for adler32 (adler32-zlib) [ 9726.046840] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [ 9726.348262] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9728.127449] Key type lgssc registered [ 9729.939704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9774.467419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9774.489796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.768176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9775.785690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9775.848292] Lustre: lustre-MDT0000: new disk, initializing [ 9775.926132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9775.945934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9781.008360] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9791.993898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.132450] Lustre: 286056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9792.189908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9792.194189] Lustre: Skipped 1 previous similar message [ 9792.315983] Lustre: lustre-MDT0001: new disk, initializing [ 9792.462835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9792.527585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9792.549012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9797.731967] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9807.414601] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9813.307049] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9820.584244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9820.914258] Lustre: lustre-OST0000: new disk, initializing [ 9820.922391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9820.931847] Lustre: 288024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9821.031211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.209798] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9830.511376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9830.528246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9830.626689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9839.031658] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9844.877757] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9845.099212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9849.729215] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9849.926827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9856.493380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9856.495352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9856.507768] LustreError: Skipped 1 previous similar message [ 9856.508541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9856.546795] Lustre: Skipped 1 previous similar message [ 9858.243312] Lustre: server umount lustre-OST0000 complete [ 9864.171078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9864.178971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.197211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9864.201312] Lustre: Skipped 1 previous similar message [ 9866.720265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9866.724667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.349842] Lustre: server umount lustre-MDT0000 complete [ 9872.857709] LustreError: 286050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145238 with bad export cookie 578680119139991968 [ 9872.873868] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9872.880487] LustreError: 286050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9873.335920] Lustre: server umount lustre-MDT0001 complete [ 9883.335302] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:47:27 (1789145247) [ 9890.149492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.664620] LustreError: 290436:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.676238] LustreError: 290436:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9890.722997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9895.905688] LustreError: 290437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.067924] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9900.003479] LustreError: 290436:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.121157] LustreError: 290437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.337463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.750512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9911.087916] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9919.777950] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9926.581659] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9934.686891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9935.230917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9941.932348] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9950.476991] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9960.933538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9960.947696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9960.970240] Lustre: Skipped 1 previous similar message [ 9960.970703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9961.004328] Lustre: Skipped 2 previous similar messages [ 9965.360595] Lustre: server umount lustre-OST0000 complete [ 9971.181060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9971.191492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9971.195794] Lustre: Skipped 1 previous similar message [ 9977.036234] Lustre: server umount lustre-MDT0000 complete [ 9981.416949] LustreError: 290433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9981.471920] LustreError: 290433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9982.930434] LustreError: 290418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145348 with bad export cookie 578680119139992850 [ 9982.939135] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9982.973092] LustreError: 290418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9983.298251] Lustre: server umount lustre-MDT0001 complete [ 9999.196799] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10004.024821] Key type lgssc unregistered [10004.374860] LNet: 293975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10004.389706] LNetError: 293975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10004.426068] LNet: Removed LNI 192.168.202.136@tcp [10005.663111] Key type .llcrypt unregistered [10005.667089] Key type ._llcrypt unregistered [10037.956644] Key type ._llcrypt registered [10037.958786] Key type .llcrypt registered [10038.141657] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10058.151514] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10059.581735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10059.913561] alg: No test for adler32 (adler32-zlib) [10061.436678] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [10061.841223] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10063.768875] Key type lgssc registered [10065.985906] Lustre: Echo OBD driver; http://www.lustre.org/ [10119.653775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10119.692698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.210881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10121.262858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10121.370431] Lustre: lustre-MDT0000: new disk, initializing [10121.549577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10121.571985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10126.441225] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10138.830795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10138.998301] Lustre: 297805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10139.070652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10139.074610] Lustre: Skipped 1 previous similar message [10139.142783] Lustre: lustre-MDT0001: new disk, initializing [10139.298961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10139.345299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10139.358872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10143.825226] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10154.766551] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10161.058916] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10169.842506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10170.175680] Lustre: lustre-OST2710: new disk, initializing [10170.183299] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [10170.191752] Lustre: 299774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10170.293100] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [10178.362629] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10178.760500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [10178.794586] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [10179.082396] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [10189.577823] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10198.324565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10198.516452] Lustre: lustre-OST03e8: new disk, initializing [10198.519934] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [10198.526142] Lustre: 300837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10198.629833] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10205.023535] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10207.297302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10207.315340] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10207.392893] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [10212.887417] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10220.995330] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10221.315264] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10226.367448] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10226.607274] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10232.430277] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10232.725163] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10238.402821] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10238.641706] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10243.567294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.578423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10243.585430] Lustre: Skipped 2 previous similar messages [10243.589224] Lustre: Skipped 1 previous similar message [10248.672912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10248.692912] Lustre: Skipped 5 previous similar messages [10249.654354] Lustre: server umount lustre-MDT0000 complete [10253.660210] LustreError: 298602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145619 with bad export cookie 7348351787311968930 [10253.678472] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10253.699831] LustreError: 298602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10253.798006] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10253.805633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10253.813162] Lustre: Skipped 2 previous similar messages [10253.819698] Lustre: Skipped 1 previous similar message [10254.087932] Lustre: server umount lustre-MDT0001 complete [10260.713618] Lustre: server umount lustre-OST2710 complete [10264.737316] Lustre: server umount lustre-OST03e8 complete [10284.284995] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10295.667628] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10338.193224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10338.429451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10338.467275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10338.576075] Lustre: lustre-MDT0000: new disk, initializing [10338.663611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10338.697920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10344.670439] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10357.446555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.655877] Lustre: 306091:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10357.782230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10357.789465] Lustre: Skipped 1 previous similar message [10357.946516] Lustre: lustre-MDT0001: new disk, initializing [10358.112546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10358.219192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10358.238801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10364.119690] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10373.180467] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10379.447655] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10386.473051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10386.659323] Lustre: lustre-OST0000: new disk, initializing [10386.666293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10386.676954] Lustre: 308057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10386.797336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10392.195930] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10394.710653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10394.725421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10394.858089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10401.686763] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10407.665359] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10407.983698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10412.513975] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10412.675900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10415.587326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10415.591807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10415.621219] Lustre: Skipped 1 previous similar message [10415.649310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10420.719162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10420.726860] Lustre: Skipped 2 previous similar messages [10420.913669] Lustre: server umount lustre-OST0000 complete [10425.826117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10425.839972] Lustre: Skipped 1 previous similar message [10430.946764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.962460] Lustre: Skipped 3 previous similar messages [10431.319413] Lustre: server umount lustre-MDT0000 complete [10435.978168] LustreError: 306082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145801 with bad export cookie 7348351787311970610 [10435.987153] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.003231] LustreError: 306082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10436.444990] Lustre: server umount lustre-MDT0001 complete [10445.361711] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:56:50 (1789145810) [10446.833935] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10448.763620] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:56:53 (1789145813) [10453.108652] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10484.910354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.264817] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10485.284670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10485.463915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10489.940882] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10498.738431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.896491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10498.919319] Lustre: 311916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10503.921726] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10527.804314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10528.159804] Lustre: Denying initial registration attempt from nid 192.168.202.136@tcp, specified as failover [10528.172769] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10528.191822] LustreError: 313368:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10528.206232] LustreError: 313368:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10528.214569] LustreError: 313368:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10528.333623] Lustre: server umount lustre-OST0000 complete [10528.340852] LustreError: 313368:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10536.928921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10536.941864] Lustre: Skipped 1 previous similar message [10536.944329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10536.953186] Lustre: Skipped 1 previous similar message [10542.509384] Lustre: server umount lustre-MDT0000 complete [10546.133954] LustreError: 311908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789145912 with bad export cookie 7348351787311971492 [10546.138919] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10546.157881] LustreError: 311908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10546.468431] Lustre: server umount lustre-MDT0001 complete [10559.737023] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10562.990326] Key type lgssc unregistered [10563.388677] LNet: 314494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10563.393403] LNetError: 314494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10563.416651] LNet: Removed LNI 192.168.202.136@tcp [10564.524814] Key type .llcrypt unregistered [10564.528582] Key type ._llcrypt unregistered [10578.789779] Key type ._llcrypt registered [10578.793801] Key type .llcrypt registered [10578.936766] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10580.566580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10580.619264] alg: No test for adler32 (adler32-zlib) [10581.748646] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [10582.235398] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10583.914407] Key type lgssc registered [10585.468723] Lustre: Echo OBD driver; http://www.lustre.org/ [10636.089962] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10645.188600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10645.215194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10646.469874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10646.498387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10646.655818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10651.747500] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10660.088783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10660.144738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10660.183845] Lustre: 317258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10660.380596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10664.413446] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10672.206778] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10677.741894] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10697.839246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10698.039379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10698.271087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10704.928714] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10713.765839] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10720.748203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10720.751843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10720.754838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10720.764821] Lustre: Skipped 1 previous similar message [10720.786644] Lustre: Skipped 1 previous similar message [10722.087151] Lustre: server umount lustre-OST0000 complete [10726.882140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10726.886119] LustreError: Skipped 1 previous similar message [10726.888296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10726.905910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10730.977990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10730.983113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10732.707139] Lustre: server umount lustre-MDT0000 complete [10737.450363] LustreError: 317251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146103 with bad export cookie 1469665659423838200 [10737.465328] LustreError: 317251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10737.465657] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10737.863399] Lustre: server umount lustre-MDT0001 complete [10751.324838] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10755.802833] Key type lgssc unregistered [10756.222366] LNet: 320849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10756.245800] LNetError: 320849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10756.294389] LNet: Removed LNI 192.168.202.136@tcp [10757.394862] Key type .llcrypt unregistered [10757.400538] Key type ._llcrypt unregistered [10787.262087] Key type ._llcrypt registered [10787.263792] Key type .llcrypt registered [10787.359816] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10788.543757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10788.654791] alg: No test for adler32 (adler32-zlib) [10789.747959] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [10790.047121] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10791.713047] Key type lgssc registered [10793.080037] Lustre: Echo OBD driver; http://www.lustre.org/ [10801.575428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10801.617993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.255965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10808.109603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10813.408828] LustreError: 322269:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.436148] LustreError: 322269:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10816.992114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.429609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10822.330716] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10830.638376] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10837.749696] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10847.217203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10847.629437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10853.700830] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10861.316654] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10873.320584] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.325555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10873.340276] Lustre: Skipped 2 previous similar messages [10873.350281] Lustre: Skipped 2 previous similar messages [10874.712200] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [10877.112867] Lustre: server umount lustre-MDT0000 complete [10878.434328] LustreError: 322269:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10878.467079] LustreError: 322269:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10879.798551] LustreError: 323910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.340489] LustreError: 322250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146247 with bad export cookie 15414622739511623179 [10881.356961] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10883.553664] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.553729] LustreError: 322263:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.569114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10886.561284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10886.563813] Lustre: Skipped 1 previous similar message [10888.021248] Lustre: server umount lustre-MDT0001 complete [10895.080043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10904.717347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10906.087904] LustreError: 321577:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa659ad1 x1876055719714432/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10906.370397] LustreError: 324099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10906.393335] LustreError: 324099:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10906.442526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10906.522624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10911.170662] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10916.714668] LustreError: 324100:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10916.746699] LustreError: 324100:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10916.752400] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10919.037337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10919.425542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10924.521771] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10925.638357] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10935.751210] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10938.032373] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10946.285848] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10948.231683] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10957.797624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10957.803962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.817199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10960.373044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.384568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10962.407046] Lustre: server umount lustre-OST0000 complete [10970.594071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10970.594833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10970.609597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.626603] Lustre: Skipped 1 previous similar message [10970.635105] Lustre: Skipped 3 previous similar messages [10972.369354] Lustre: server umount lustre-MDT0000 complete [10975.717363] LustreError: 325867:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.814373] LustreError: 322248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146342 with bad export cookie 15414622739511630788 [10976.830946] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10976.846440] LustreError: 322248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10977.302604] Lustre: server umount lustre-MDT0001 complete [10989.152611] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10991.576934] Key type lgssc unregistered [10991.755956] LNet: 328542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10991.760774] LNetError: 328542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10992.807785] LNet: Removed LNI 192.168.202.136@tcp [10993.745473] Key type .llcrypt unregistered [10993.748177] Key type ._llcrypt unregistered [11019.631947] Key type ._llcrypt registered [11019.633097] Key type .llcrypt registered [11019.739534] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11020.697675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11020.721288] alg: No test for adler32 (adler32-zlib) [11021.812499] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [11022.114833] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11023.775925] Key type lgssc registered [11025.056344] Lustre: Echo OBD driver; http://www.lustre.org/ [11034.065641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11034.090171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.574220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11040.591312] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11045.862177] LustreError: 329959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.891979] LustreError: 329959:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11049.761497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.187229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11055.528896] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11064.416174] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11072.209340] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11076.064240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11076.070163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11076.093375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11080.161157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11080.164817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.284335] Lustre: server umount lustre-MDT0000 complete [11085.281220] LustreError: 329958:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11085.290989] LustreError: 329958:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11086.539782] LustreError: 329941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146452 with bad export cookie 4327872405655262993 [11086.568413] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11086.570264] LustreError: 329941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11086.990281] Lustre: server umount lustre-MDT0001 complete [11103.151710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11103.474952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11103.507234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11103.738554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11109.025821] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11118.487852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11118.598658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11118.642506] Lustre: 332548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11118.945355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11124.065455] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11132.949053] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11139.865518] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11150.479307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11150.755562] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11150.771549] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11150.782473] LustreError: 334435:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [11150.792607] LustreError: 334435:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [11150.806521] LustreError: 334435:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11150.891354] Lustre: server umount lustre-OST0000 complete [11150.893213] LustreError: 334435:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [11163.121966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.463929] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11163.654635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11170.694572] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11173.032650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11180.068394] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11188.368231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11188.448722] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11188.518869] Lustre: lustre-OST0001: new disk, initializing [11188.522336] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11188.526658] Lustre: 335975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11188.624111] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11194.979500] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11196.026683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11196.057233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11196.220049] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11202.547368] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11206.626069] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11206.631146] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.636845] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11206.653611] Lustre: Skipped 1 previous similar message [11206.657882] Lustre: Skipped 1 previous similar message [11211.753833] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11211.761186] Lustre: Skipped 1 previous similar message [11211.941085] Lustre: server umount lustre-OST0001 complete [11216.864936] LustreError: 334901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11216.866819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11216.871443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11216.882194] LustreError: 334901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11216.931957] Lustre: Skipped 1 previous similar message [11221.998882] LustreError: 335366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11222.019784] LustreError: 335366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11222.290933] Lustre: server umount lustre-OST0000 complete [11226.608584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11226.620554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.655379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11226.661789] Lustre: Skipped 3 previous similar messages [11232.625727] Lustre: server umount lustre-MDT0000 complete [11236.651719] LustreError: 332540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146602 with bad export cookie 4327872405655263546 [11236.664700] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11237.269554] Lustre: server umount lustre-MDT0001 complete [11248.533443] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11251.292450] Key type lgssc unregistered [11251.615378] LNet: 337937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11251.621933] LNetError: 337937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11251.632984] LNet: Removed LNI 192.168.202.136@tcp [11252.553422] Key type .llcrypt unregistered [11252.556785] Key type ._llcrypt unregistered [11329.721788] Key type ._llcrypt registered [11329.723867] Key type .llcrypt registered [11329.884468] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [11342.187865] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11343.170138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11343.184866] alg: No test for adler32 (adler32-zlib) [11344.256162] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [11344.494905] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11346.159182] Key type lgssc registered [11347.154614] Lustre: Echo OBD driver; http://www.lustre.org/ [11377.147542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11377.189805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.410341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11378.430291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11378.516394] Lustre: lustre-MDT0000: new disk, initializing [11378.625260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11378.647452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11383.612242] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11393.894158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.982383] Lustre: 342850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11394.034243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11394.038328] Lustre: Skipped 1 previous similar message [11394.101439] Lustre: lustre-MDT0001: new disk, initializing [11394.172194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11394.206048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11394.211398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11398.422395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11407.685836] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11413.368850] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11420.750487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.104544] Lustre: lustre-OST0000: new disk, initializing [11421.107870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11421.114201] Lustre: 344819:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11421.220554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11427.622520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11427.951157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11427.965504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11428.127116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11437.691052] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11442.685962] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11442.831445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11448.556134] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11448.900309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11453.920554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11453.942828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11453.972139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11457.653651] Lustre: server umount lustre-OST0000 complete [11464.163817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11464.170406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11464.179406] Lustre: Skipped 2 previous similar messages [11464.202992] Lustre: Skipped 2 previous similar messages [11469.281722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11469.298163] Lustre: Skipped 1 previous similar message [11470.026629] Lustre: server umount lustre-MDT0000 complete [11474.747993] LustreError: 342844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789146840 with bad export cookie 53808809319031275 [11474.758306] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11474.759778] LustreError: 342844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11475.247819] Lustre: server umount lustre-MDT0001 complete [11484.919805] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:14:09 (1789146849) [11492.641135] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:14:17 (1789146857) [11500.439699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11500.971697] LustreError: 347577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11500.997483] LustreError: 347577:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11501.040941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11505.976779] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11506.162516] LustreError: 347577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.174856] LustreError: 347577:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11511.264906] LustreError: 347578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11514.459747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11514.845150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11519.681969] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11527.155934] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11533.810973] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11542.442465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11543.170823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11549.940460] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11557.016157] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11567.351422] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11569.486257] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11571.228898] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11573.267138] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11575.255445] Lustre: Failing over lustre-MDT0000 [11575.274336] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11575.294767] Lustre: Skipped 1 previous similar message [11575.298204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11575.312124] Lustre: Skipped 2 previous similar messages [11575.427602] Lustre: server umount lustre-MDT0000 complete [11579.215112] LustreError: 347572:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.303767] LustreError: 347574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.328318] LustreError: 347574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11594.354036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11594.507198] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11594.546206] LustreError: 347573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.554991] LustreError: 347573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11594.782857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11594.842127] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11599.666640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11599.848781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11599.884703] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11599.932043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11600.058560] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11610.671668] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11612.914272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11616.743748] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11618.622811] Lustre: Failing over lustre-MDT0000 [11618.837678] Lustre: server umount lustre-MDT0000 complete [11620.179483] LustreError: 347572:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.328354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11620.328690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11620.341782] LustreError: Skipped 1 previous similar message [11620.360390] Lustre: Skipped 3 previous similar messages [11636.704838] Lustre: 341076:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789146986/real 1789146986] req@00000000c11203e5 x1876056300677504/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789147002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11636.755833] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11637.535521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.946912] LustreError: 341073:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006dff0f7b x1876056300683136/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11647.404033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11647.509145] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11651.136335] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11651.887808] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11652.586211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11652.588137] Lustre: Skipped 2 previous similar messages [11652.615698] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11652.679873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11660.431747] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11661.977240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11673.063194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11673.064539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.079307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11673.101777] Lustre: Skipped 2 previous similar messages [11673.122815] Lustre: Skipped 2 previous similar messages [11675.772500] Lustre: server umount lustre-MDT0000 complete [11678.179131] LustreError: 347574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.208365] LustreError: 347574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 28 previous similar messages [11680.735531] LustreError: 347558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147046 with bad export cookie 53808809319033998 [11680.738664] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11680.747564] LustreError: 347558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11681.239578] Lustre: server umount lustre-MDT0001 complete [11694.792423] Lustre: server umount lustre-OST0000 complete [11711.841639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11712.190371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11717.172881] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11727.282922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.035750] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11743.399199] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11748.780892] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11753.389240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11757.772298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11758.491928] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11768.534228] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11782.628588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11782.643068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11782.659373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11782.675096] Lustre: Skipped 1 previous similar message [11785.594991] Lustre: server umount lustre-OST0000 complete [11795.820297] Lustre: server umount lustre-MDT0000 complete [11797.988798] LustreError: 354643:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11798.030661] LustreError: 354643:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11799.709619] LustreError: 353896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147165 with bad export cookie 53808809319034733 [11799.711154] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11799.721887] LustreError: 353896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11800.196284] Lustre: server umount lustre-MDT0001 complete [11810.272391] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11812.915710] Key type lgssc unregistered [11813.168844] LNet: 357452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11813.172499] LNetError: 357452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11813.180572] LNet: Removed LNI 192.168.202.136@tcp [11814.073852] Key type .llcrypt unregistered [11814.075959] Key type ._llcrypt unregistered [11844.190901] Key type ._llcrypt registered [11844.195995] Key type .llcrypt registered [11844.321024] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11845.276309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11845.434064] alg: No test for adler32 (adler32-zlib) [11846.577848] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [11846.876476] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11848.543276] Key type lgssc registered [11849.831170] Lustre: Echo OBD driver; http://www.lustre.org/ [11858.641442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11858.667604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11860.326555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11864.966752] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11870.694771] LustreError: 358974:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11870.718084] LustreError: 358974:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11874.421868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.831841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11880.087316] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11888.296670] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11894.454721] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11902.781770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.634237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11905.651089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11911.309540] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11917.849217] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11930.599463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.614752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.635493] Lustre: Skipped 1 previous similar message [11934.894869] Lustre: server umount lustre-MDT0000 complete [11935.725966] LustreError: 359944:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.761736] LustreError: 359944:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11938.809988] LustreError: 361552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147304 with bad export cookie 11672596298322221787 [11938.816700] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11938.820759] LustreError: 361552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11939.172970] Lustre: server umount lustre-MDT0001 complete [11953.560044] Lustre: server umount lustre-OST0000 complete [11969.282270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11969.826876] LustreError: 362684:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11969.930932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11974.969928] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11975.139452] LustreError: 362685:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.236330] LustreError: 362684:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11984.737451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.291992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11990.172658] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11999.115785] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12005.706331] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12013.141731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12013.600608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12014.658083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12020.443469] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12028.608815] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12041.195700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.202282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.223267] Lustre: Skipped 2 previous similar messages [12041.244811] Lustre: Skipped 4 previous similar messages [12046.307039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.316924] Lustre: Skipped 2 previous similar messages [12046.784468] Lustre: server umount lustre-MDT0000 complete [12050.976920] LustreError: 362709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147416 with bad export cookie 11672596298322222900 [12050.981994] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12050.993539] LustreError: 362709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12051.211193] Lustre: server umount lustre-MDT0001 complete [12065.695988] Lustre: server umount lustre-OST0000 complete [12081.956959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.497637] LustreError: 366596:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.531217] LustreError: 366596:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12082.612052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12087.136608] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12095.745978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.728646] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12108.983368] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12115.541302] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12124.192715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12124.619333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12124.624843] Lustre: Skipped 1 previous similar message [12127.742636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12132.211943] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12139.528629] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12146.149879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.159673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12146.164745] Lustre: Skipped 2 previous similar messages [12146.173484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12149.656052] Lustre: server umount lustre-OST0000 complete [12156.384394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.418323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12156.430661] Lustre: Skipped 2 previous similar messages [12161.511895] LustreError: 366591:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.529466] LustreError: 366591:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12163.389489] Lustre: server umount lustre-MDT0000 complete [12168.146411] LustreError: 366577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147534 with bad export cookie 11672596298322223922 [12168.156501] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12168.177812] LustreError: 366577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12168.795924] Lustre: server umount lustre-MDT0001 complete [12183.442410] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12186.359848] Key type lgssc unregistered [12186.580421] LNet: 370137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12186.592331] LNetError: 370137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12186.622107] LNet: Removed LNI 192.168.202.136@tcp [12187.757651] Key type .llcrypt unregistered [12187.761697] Key type ._llcrypt unregistered [12216.082042] Key type ._llcrypt registered [12216.083494] Key type .llcrypt registered [12216.191160] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12217.526091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12217.574611] alg: No test for adler32 (adler32-zlib) [12218.666744] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [12218.967362] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12220.639585] Key type lgssc registered [12222.152499] Lustre: Echo OBD driver; http://www.lustre.org/ [12231.942703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12231.955370] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.965848] LustreError: 371581:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12231.989202] LustreError: 371581:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12231.992380] LustreError: 371581:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12232.003523] LustreError: 371581:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12232.018884] LustreError: 371581:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12241.174263] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12241.188904] LustreError: 371980:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12241.219959] LustreError: 371980:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12241.228332] LustreError: 371980:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12241.234864] LustreError: 371980:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12241.241326] LustreError: 371980:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12264.925622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12268.683965] Key type lgssc unregistered [12269.124560] LNet: 372987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12269.140571] LNetError: 372987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12269.166819] LNet: Removed LNI 192.168.202.136@tcp [12270.191384] Key type .llcrypt unregistered [12270.193691] Key type ._llcrypt unregistered [12289.349629] Key type ._llcrypt registered [12289.351900] Key type .llcrypt registered [12289.434896] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [12307.920269] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12309.297921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12309.467270] alg: No test for adler32 (adler32-zlib) [12310.683886] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [12310.992659] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12312.695225] Key type lgssc registered [12313.883384] Lustre: Echo OBD driver; http://www.lustre.org/ [12355.326389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12355.363114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12356.663836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12356.698407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12356.763787] Lustre: lustre-MDT0000: new disk, initializing [12356.854955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12356.877424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12362.440825] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12374.394554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12374.518326] Lustre: 376142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12374.593909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12374.598923] Lustre: Skipped 1 previous similar message [12374.702678] Lustre: lustre-MDT0001: new disk, initializing [12374.852785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12374.919628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12374.942017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12379.727817] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12390.268068] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12396.551942] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12404.444684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12404.683914] Lustre: lustre-OST0000: new disk, initializing [12404.688450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12404.695978] Lustre: 378113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12404.788122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12413.013319] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12413.069844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12413.081410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12413.205855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12425.440379] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12430.573293] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12430.874931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12436.559935] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12436.815353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12439.009366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12439.012807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12439.036294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12439.052050] Lustre: Skipped 1 previous similar message [12439.082801] Lustre: Skipped 1 previous similar message [12444.140043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12444.157067] Lustre: Skipped 1 previous similar message [12449.254765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12449.272743] Lustre: Skipped 1 previous similar message [12453.344055] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12453.922468] Lustre: server umount lustre-OST0000 complete [12462.049260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12462.052521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12462.060080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12464.626533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12469.728682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12469.749619] Lustre: Skipped 3 previous similar messages [12474.335192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12474.661169] Lustre: server umount lustre-MDT0000 complete [12478.842587] LustreError: 376136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789147844 with bad export cookie 5826951474509331389 [12478.844575] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12478.848955] LustreError: 376136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12479.157365] Lustre: server umount lustre-MDT0001 complete [12490.011171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12492.536552] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:30:56 (1789147856) [12500.968319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.433224] LustreError: 380576:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12501.457252] LustreError: 380576:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12501.523829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12506.593728] LustreError: 380577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.800564] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12511.714472] LustreError: 380576:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.835286] LustreError: 380577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12518.473150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.939740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12524.799455] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12533.701413] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12540.960491] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12551.663581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12552.225438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12560.130264] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12571.078558] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12579.237885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12579.393101] Lustre: lustre-OST0001: new disk, initializing [12579.396966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12579.402247] Lustre: 383416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12579.478374] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12587.237612] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12589.889466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12589.912942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12590.050705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12600.118831] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12605.433863] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12605.444162] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12605.458236] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12605.488989] Lustre: Skipped 1 previous similar message [12610.954917] Lustre: server umount lustre-OST0001 complete [12612.424142] LustreError: 384227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.785313] LustreError: 384227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.812413] LustreError: 384227:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12631.017131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12631.025048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12631.066041] Lustre: Skipped 1 previous similar message [12631.091652] Lustre: Skipped 5 previous similar messages [12636.831465] Lustre: server umount lustre-OST0000 complete [12646.370492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12650.040102] Lustre: server umount lustre-MDT0000 complete [12651.497581] LustreError: 382140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.536924] LustreError: 382140:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [12656.019677] LustreError: 382422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148022 with bad export cookie 5826951474509332278 [12656.036424] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12656.042280] LustreError: 382422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12662.438713] Lustre: server umount lustre-MDT0001 complete [12673.972762] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12677.112940] Key type lgssc unregistered [12677.476761] LNet: 385411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12677.491527] LNetError: 385411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12677.516198] LNet: Removed LNI 192.168.202.136@tcp [12678.479406] Key type .llcrypt unregistered [12678.482147] Key type ._llcrypt unregistered [12738.547808] Key type ._llcrypt registered [12738.556827] Key type .llcrypt registered [12738.759352] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12740.386452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12740.445180] alg: No test for adler32 (adler32-zlib) [12741.583521] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [12741.884724] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12743.623906] Key type lgssc registered [12745.201609] Lustre: Echo OBD driver; http://www.lustre.org/ [12754.781308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12754.803053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12756.057769] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12756.086301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12756.276507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12761.771347] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12772.414970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12772.527401] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12772.594321] Lustre: 388211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12772.976603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12778.576599] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12788.028816] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12793.877797] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12798.436887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12798.455425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12798.479717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12802.319816] Lustre: server umount lustre-MDT0000 complete [12805.695890] LustreError: 388203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148171 with bad export cookie 13637875991631283861 [12805.700252] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12805.708863] LustreError: 388203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12806.073598] Lustre: server umount lustre-MDT0001 complete [12816.074921] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:36:20 (1789148180) [12823.286639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12823.720882] LustreError: 390725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12823.741706] LustreError: 390725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12823.787141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12829.158397] LustreError: 390726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12829.256736] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12834.279304] LustreError: 390725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.353421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12839.407344] LustreError: 390726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12839.879945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12845.290797] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12853.922171] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12859.852981] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12867.981915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12868.154676] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12868.386939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12874.067966] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12883.368978] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12892.922258] Lustre: Permanently deactivating lustre-OST0000 [12892.933563] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12900.349869] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12901.569462] LustreError: 393663:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12901.581217] LustreError: 393663:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12901.587185] LustreError: 393663:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12905.936676] LustreError: 393713:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12905.939264] LustreError: 393713:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12914.403480] Lustre: server umount lustre-OST0000 complete [12917.093650] LustreError: 393963:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12917.100042] LustreError: 393963:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12918.753602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12918.758461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12918.765178] Lustre: Skipped 2 previous similar messages [12918.767493] Lustre: Skipped 1 previous similar message [12923.874563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12923.883220] Lustre: Skipped 2 previous similar messages [12928.999467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12929.011205] Lustre: Skipped 1 previous similar message [12929.119098] Lustre: server umount lustre-MDT0000 complete [12933.667463] LustreError: 390707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148299 with bad export cookie 13637875991631284435 [12933.675693] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12933.678224] LustreError: 390707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12944.087986] Lustre: server umount lustre-MDT0001 complete [12954.311887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12960.034615] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12963.585428] LustreError: 395123:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12971.040655] Lustre: 395462:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.136@tcp [12973.292801] Lustre: server umount MGS complete [12986.846894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12987.286911] LustreError: 396186:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12987.314375] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12987.320285] Lustre: Skipped 1 previous similar message [12987.373553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12992.887756] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12997.634115] LustreError: 396187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12997.660714] LustreError: 396187:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13001.251307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13001.558915] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13005.862035] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13015.505721] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13022.830981] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13025.834686] Lustre: Permanently reactivating lustre-OST0000 [13025.873288] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [13025.884350] Lustre: Skipped 2 previous similar messages [13055.456185] LustreError: 398009:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13063.418271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13063.862408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13063.866068] Lustre: Skipped 1 previous similar message [13071.962061] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13080.215526] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13086.723975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13086.740396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13086.744968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13086.780804] Lustre: Skipped 1 previous similar message [13086.787550] LustreError: Skipped 1 previous similar message [13088.365343] Lustre: server umount lustre-OST0000 complete [13096.939035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13096.947330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13096.956045] Lustre: Skipped 1 previous similar message [13096.975314] Lustre: Skipped 2 previous similar messages [13100.478845] Lustre: server umount lustre-MDT0000 complete [13102.056238] LustreError: 396181:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.075930] LustreError: 396181:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13105.064994] LustreError: 396166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148471 with bad export cookie 13637875991631285485 [13105.070886] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13105.088307] LustreError: 396166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13105.491392] Lustre: server umount lustre-MDT0001 complete [13118.683487] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13122.124957] Key type lgssc unregistered [13122.385926] LNet: 400603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13122.391937] LNetError: 400603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13122.413446] LNet: Removed LNI 192.168.202.136@tcp [13123.519710] Key type .llcrypt unregistered [13123.521655] Key type ._llcrypt unregistered [13144.708681] Key type ._llcrypt registered [13144.711150] Key type .llcrypt registered [13144.807926] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13161.074682] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13162.199111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13162.451991] alg: No test for adler32 (adler32-zlib) [13163.639988] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [13164.010812] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13165.743282] Key type lgssc registered [13166.879666] Lustre: Echo OBD driver; http://www.lustre.org/ [13205.484027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13205.518773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13206.756531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13206.784728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13206.866792] Lustre: lustre-MDT0000: new disk, initializing [13206.923816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13206.958455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13212.012599] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13224.245967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13224.469578] Lustre: 403764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13224.554973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13224.559953] Lustre: Skipped 1 previous similar message [13224.693748] Lustre: lustre-MDT0001: new disk, initializing [13224.848794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13224.946106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13224.952322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13229.461711] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13242.397345] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13252.111291] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13262.178358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13262.689185] Lustre: lustre-OST0000: new disk, initializing [13262.693445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13262.699502] Lustre: 405732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13262.820654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13269.915669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13272.175725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13272.200286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13272.331682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13279.949550] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13286.071497] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13286.406570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13292.742090] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13292.956944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13298.157293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13298.159202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13298.165120] LustreError: Skipped 1 previous similar message [13298.166184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13298.200162] Lustre: Skipped 1 previous similar message [13303.265147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13303.284158] Lustre: Skipped 1 previous similar message [13308.389184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13308.394210] Lustre: Skipped 2 previous similar messages [13309.410136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13309.781967] Lustre: server umount lustre-OST0000 complete [13317.089726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13317.112077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13317.146715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13318.630049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13321.513041] Lustre: server umount lustre-MDT0000 complete [13326.203066] LustreError: 403753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148692 with bad export cookie 4672918962986274827 [13326.209776] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13326.218797] LustreError: 403753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13326.584686] Lustre: server umount lustre-MDT0001 complete [13336.367386] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13338.251601] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:45:03 (1789148703) [13345.576825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13345.944213] LustreError: 408193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13345.964342] LustreError: 408193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13346.040448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13351.396373] LustreError: 408194:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13351.411593] LustreError: 408194:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13351.530399] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13355.502819] LustreError: 408193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13360.615906] LustreError: 408194:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13361.586929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13362.005827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13368.655349] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13377.201108] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13384.599767] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13392.452864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13392.925062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13401.201650] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13410.263703] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13412.576506] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13419.680391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13425.635204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13425.654604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13425.662428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13425.676662] Lustre: Skipped 1 previous similar message [13425.694810] Lustre: Skipped 3 previous similar messages [13431.221953] Lustre: server umount lustre-OST0000 complete [13435.872237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13435.877321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13435.891158] Lustre: Skipped 1 previous similar message [13435.916375] Lustre: Skipped 3 previous similar messages [13441.844851] Lustre: server umount lustre-MDT0000 complete [13446.117528] LustreError: 408193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13446.137356] LustreError: 408193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13446.175473] LustreError: 408174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148812 with bad export cookie 4672918962986275709 [13446.186223] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13446.570209] Lustre: server umount lustre-MDT0001 complete [13460.053442] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13463.531734] Key type lgssc unregistered [13463.939659] LNet: 411923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13463.958382] LNetError: 411923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13464.997444] LNet: Removed LNI 192.168.202.136@tcp [13465.972986] Key type .llcrypt unregistered [13465.978809] Key type ._llcrypt unregistered [13494.404423] Key type ._llcrypt registered [13494.405840] Key type .llcrypt registered [13494.563181] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13494.905846] Key type .llcrypt unregistered [13494.910752] Key type ._llcrypt unregistered [13515.154862] Key type ._llcrypt registered [13515.157718] Key type .llcrypt registered [13515.276550] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13516.369299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13516.547286] alg: No test for adler32 (adler32-zlib) [13517.751697] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [13518.138356] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13519.944978] Key type lgssc registered [13521.335283] Lustre: Echo OBD driver; http://www.lustre.org/ [13530.282743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13530.305264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13531.774580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13536.550253] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13541.352813] LustreError: 414146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.404350] LustreError: 414146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13546.471794] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13547.900218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13548.204865] LustreError: 414146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.228127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13548.232485] LustreError: 414146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13553.641833] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.676092] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13554.934970] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13556.760868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13561.825898] LustreError: 414890:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13561.861646] LustreError: 414890:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13566.522159] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13572.074111] LustreError: 414146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13572.116441] LustreError: 414146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13578.381536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13579.004816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13584.884976] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13595.155604] Lustre: server umount lustre-OST0000 complete [13599.713558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13599.731908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13599.735268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13599.746210] Lustre: Skipped 1 previous similar message [13599.757662] Lustre: Skipped 1 previous similar message [13604.852164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13604.866296] Lustre: Skipped 1 previous similar message [13609.148335] Lustre: server umount lustre-MDT0000 complete [13609.961237] LustreError: 416255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13610.002703] LustreError: 416255:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13613.834949] LustreError: 414128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789148979 with bad export cookie 9266160754939678471 [13613.844765] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13613.855508] LustreError: 414128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13623.500631] Lustre: server umount lustre-MDT0001 complete [13637.141463] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13640.711544] Key type lgssc unregistered [13641.088823] LNet: 417301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13641.102499] LNetError: 417301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13642.150303] LNet: Removed LNI 192.168.202.136@tcp [13643.414247] Key type .llcrypt unregistered [13643.419114] Key type ._llcrypt unregistered [13672.957720] Key type ._llcrypt registered [13672.960714] Key type .llcrypt registered [13673.088954] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13674.445563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13674.463585] alg: No test for adler32 (adler32-zlib) [13675.761318] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [13676.155595] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13677.936886] Key type lgssc registered [13678.998954] Lustre: Echo OBD driver; http://www.lustre.org/ [13688.148639] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13688.175744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13703.840213] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13708.959368] LustreError: 418693:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [13735.711176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13744.027211] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13754.399766] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13763.783144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13764.587875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13764.685348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13769.700877] LustreError: 418718:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.721891] LustreError: 418718:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13769.875677] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13770.731068] LustreError: 419730:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13770.760909] LustreError: 419730:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13775.855594] LustreError: 419732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13775.876721] LustreError: 419732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13778.718586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13779.035275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13785.152964] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13793.426927] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13802.883885] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13810.657133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13810.666934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13810.694256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13812.777161] Lustre: server umount lustre-OST0000 complete [13819.873096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13819.878290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13819.881409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13819.889902] Lustre: Skipped 1 previous similar message [13823.794486] Lustre: server umount lustre-MDT0000 complete [13824.995991] LustreError: 419732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13825.011738] LustreError: 419732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13827.971584] LustreError: 418700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149193 with bad export cookie 15933699469958255740 [13827.984128] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13838.456720] Lustre: server umount lustre-MDT0001 complete [13852.504298] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13855.367456] Key type lgssc unregistered [13855.597231] LNet: 422272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13855.603053] LNetError: 422272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13855.621126] LNet: Removed LNI 192.168.202.136@tcp [13856.390800] Key type .llcrypt unregistered [13856.392079] Key type ._llcrypt unregistered [13887.702447] Key type ._llcrypt registered [13887.707507] Key type .llcrypt registered [13887.831549] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13889.229773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13889.320429] alg: No test for adler32 (adler32-zlib) [13890.439487] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [13890.761555] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13892.431167] Key type lgssc registered [13893.707972] Lustre: Echo OBD driver; http://www.lustre.org/ [13901.700180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13901.713961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13903.157512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13907.750842] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13913.568833] LustreError: 423693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13913.582919] LustreError: 423693:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13916.894801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13917.173932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13922.465694] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13930.321320] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13936.037675] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13943.374983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13943.684842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13948.862388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13951.561825] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13962.345138] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13970.791430] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [13971.884897] Lustre: server umount lustre-MDT0000 complete [13972.457153] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13972.468173] LustreError: 423692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13972.478529] Lustre: Skipped 1 previous similar message [13975.338193] LustreError: 423672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.36@tcp arrived at 1789149341 with bad export cookie 3603955469782825982 [13975.378202] LustreError: 423672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13977.568890] LustreError: 424649:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13977.603855] LustreError: 424649:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13978.248497] LustreError: 423673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149344 with bad export cookie 3603955469782825086 [13978.258461] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13982.696032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13982.701393] LustreError: 423687:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13982.704082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13982.704093] Lustre: Skipped 1 previous similar message [13982.718978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13982.803352] LustreError: 423687:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13984.649759] Lustre: server umount lustre-OST0000 complete [13988.834663] LustreError: 424649:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13988.848756] LustreError: 424649:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13993.535102] Lustre: server umount lustre-MDT0001 complete [14007.428868] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14010.380798] Key type lgssc unregistered [14010.801391] LNet: 427380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14010.814741] LNetError: 427380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14010.832327] LNet: Removed LNI 192.168.202.136@tcp [14011.927649] Key type .llcrypt unregistered [14011.930118] Key type ._llcrypt unregistered [14038.077332] Key type ._llcrypt registered [14038.080502] Key type .llcrypt registered [14038.231218] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [14040.049490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14040.263689] alg: No test for adler32 (adler32-zlib) [14041.540861] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [14041.943687] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [14043.719442] Key type lgssc registered [14046.021661] Lustre: Echo OBD driver; http://www.lustre.org/ [14055.092994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14055.102594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14056.520111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14062.801137] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14066.659177] LustreError: 428797:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14066.676736] LustreError: 428797:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14071.465964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14071.752250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14076.096158] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14085.813364] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14095.046800] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14104.680059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14105.235829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14111.429207] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14120.405298] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14126.723318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14136.134850] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [14137.314849] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14137.322983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14140.394637] Lustre: server umount lustre-MDT0001 complete [14141.232805] LustreError: 430509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14142.437601] LustreError: 430085:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14142.462197] LustreError: 430085:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14147.556665] LustreError: 428798:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14147.556862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14147.584866] Lustre: Skipped 1 previous similar message [14147.593578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14147.597383] Lustre: Skipped 1 previous similar message [14152.684871] LustreError: 428797:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14152.688048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14153.245812] Lustre: server umount lustre-OST0000 complete [14157.171109] Lustre: server umount lustre-MDT0000 complete [14170.309904] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14172.587416] Key type lgssc unregistered [14172.879664] LNet: 432511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14172.895314] LNetError: 432511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14173.933471] LNet: Removed LNI 192.168.202.136@tcp [14174.823587] Key type .llcrypt unregistered [14174.825532] Key type ._llcrypt unregistered [14198.952902] Key type ._llcrypt registered [14198.954829] Key type .llcrypt registered [14199.053247] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [14213.707915] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [14215.335565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14215.354673] alg: No test for adler32 (adler32-zlib) [14216.424945] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [14216.676454] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [14218.351457] Key type lgssc registered [14219.650905] Lustre: Echo OBD driver; http://www.lustre.org/ [14255.837971] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [14269.911155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14269.963079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14271.209651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14271.246147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14271.315452] Lustre: lustre-MDT0000: new disk, initializing [14271.427417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14271.442136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14274.768549] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14288.875826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14289.047825] Lustre: lustre-OST0000: new disk, initializing [14289.049678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14289.051382] Lustre: Skipped 1 previous similar message [14289.053626] Lustre: 437556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14289.108561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14295.661378] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14298.251077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14298.263586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14298.306177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14305.589984] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14319.365372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14319.578566] Lustre: 436535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14319.640400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14319.735957] Lustre: lustre-MDT0001: new disk, initializing [14319.861714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14319.889420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14319.903123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14324.327748] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14333.100761] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14333.358261] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14337.781509] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14337.975070] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14346.068686] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [14347.234834] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14347.252258] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14347.272410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14347.285209] Lustre: Skipped 1 previous similar message [14349.020420] Lustre: server umount lustre-MDT0001 complete [14357.475303] LustreError: 436549:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14357.493342] LustreError: 436549:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [14362.595050] LustreError: 436548:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14362.612028] LustreError: 436548:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14367.715547] LustreError: 436549:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14367.717441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14367.746150] Lustre: Skipped 1 previous similar message [14367.756619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14372.180183] Lustre: server umount lustre-OST0000 complete [14372.844857] LustreError: 436548:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14390.240548] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14390.629682] Lustre: server umount lustre-MDT0000 complete [14406.143674] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14408.757099] Key type lgssc unregistered [14409.086932] LNet: 441319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14409.100724] LNetError: 441319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14409.129551] LNet: Removed LNI 192.168.202.136@tcp [14410.172102] Key type .llcrypt unregistered [14410.175992] Key type ._llcrypt unregistered [14454.447700] Key type ._llcrypt registered [14454.453853] Key type .llcrypt registered [14454.647785] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14455.103346] Key type .llcrypt unregistered [14455.104880] Key type ._llcrypt unregistered [14475.996746] Key type ._llcrypt registered [14476.003063] Key type .llcrypt registered [14476.142807] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [14478.236732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14478.270714] alg: No test for adler32 (adler32-zlib) [14479.280573] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [14479.449116] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [14481.080049] Key type lgssc registered [14482.421244] Lustre: Echo OBD driver; http://www.lustre.org/ [14520.768579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14520.802474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14522.333087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14522.468575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14522.684143] Lustre: lustre-MDT0000: new disk, initializing [14522.853075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14522.880953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14529.745441] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14547.599864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14547.742901] Lustre: 444705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14547.825300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14547.829772] Lustre: Skipped 1 previous similar message [14547.927178] Lustre: lustre-MDT0001: new disk, initializing [14548.129136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14548.223839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14548.241667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14553.130767] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14565.309978] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14574.165751] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14583.819321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14584.084952] Lustre: lustre-OST0000: new disk, initializing [14584.088240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14584.092344] Lustre: 446677:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14584.186504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14591.952738] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14592.700989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14592.717729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14592.813337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14603.714235] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14613.476234] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14613.481641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14613.498177] Lustre: Skipped 2 previous similar messages [14613.513887] Lustre: Skipped 2 previous similar messages [14618.745270] Lustre: server umount lustre-MDT0000 complete [14623.714452] LustreError: 444714:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14623.746340] LustreError: 444714:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [14623.789666] LustreError: 444699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789149989 with bad export cookie 292337503395609160 [14623.803660] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14623.809798] LustreError: 444699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14624.228358] Lustre: server umount lustre-MDT0001 complete [14638.412708] Lustre: server umount lustre-OST0000 complete [14650.359431] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:06:55 (1789150015) [14658.089829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14658.840933] LustreError: 448708:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14658.903085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14664.164567] LustreError: 448709:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14664.234542] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14668.266487] LustreError: 448708:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14673.382819] LustreError: 448709:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14674.532358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14675.003755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14680.809918] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14691.277810] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14700.480376] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14721.158663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14721.544443] Lustre: Found index 0 for lustre-OST0000, updating log [14721.828216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14729.466568] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14730.038596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14738.323753] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14744.609465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14744.631877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14745.724649] LustreError: 450536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14745.750172] LustreError: 450536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14745.840596] Lustre: server umount lustre-OST0000 complete [14761.965472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14761.980326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14762.003497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14767.589178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14767.595209] Lustre: Skipped 3 previous similar messages [14772.715697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14772.720498] Lustre: Skipped 1 previous similar message [14776.287219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14776.490637] Lustre: server umount lustre-MDT0000 complete [14778.851805] LustreError: 448727:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14778.861895] LustreError: 448727:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14780.577755] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [14780.725123] LustreError: 448690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150146 with bad export cookie 292337503395610553 [14780.734199] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14780.746548] LustreError: 448690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14781.000623] Lustre: server umount lustre-MDT0001 complete [14790.288727] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:09:15 (1789150155) [15425.948382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15426.661831] LustreError: 455128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15426.758977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15432.131381] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15432.177907] LustreError: 455129:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15442.413107] LustreError: 455129:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15442.439920] LustreError: 455129:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15442.723461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15443.463150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15449.872431] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15458.275892] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15465.363599] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15484.063585] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [15486.835768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15487.140956] Lustre: Found index 0 for lustre-OST0000, updating log [15487.585579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15495.446869] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15504.431292] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15507.245069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15507.266352] Lustre: Skipped 1 previous similar message [15507.273474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15507.280996] Lustre: Skipped 2 previous similar messages [15512.521533] Lustre: server umount lustre-OST0000 complete [15518.242407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15518.257826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15518.264730] Lustre: Skipped 1 previous similar message [15518.275684] Lustre: Skipped 1 previous similar message [15520.358405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15520.369531] Lustre: Skipped 3 previous similar messages [15523.838453] Lustre: server umount lustre-MDT0000 complete [15524.520026] LustreError: 455128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15524.543660] LustreError: 455128:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15528.004847] LustreError: 455109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789150894 with bad export cookie 292337503395611624 [15528.008536] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15528.021903] LustreError: 455109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15528.621382] Lustre: server umount lustre-MDT0001 complete [15544.479169] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [15562.379759] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [15573.977144] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15618.372913] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:23:03 (1789150983) [15637.698781] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15641.271590] Key type lgssc unregistered [15641.634819] LNet: 461749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15641.641263] LNetError: 461749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15750.182555] LNet: Removed LNI 192.168.202.136@tcp [15751.532545] Key type .llcrypt unregistered [15751.534365] Key type ._llcrypt unregistered [15772.453200] Key type ._llcrypt registered [15772.456259] Key type .llcrypt registered [15772.593462] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15774.388828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15774.452107] alg: No test for adler32 (adler32-zlib) [15775.602850] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [15776.010459] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15777.921939] Key type lgssc registered [15779.234055] Lustre: Echo OBD driver; http://www.lustre.org/ [15785.661406] LNet: Added LNI 192.168.202.136@tcp100 [8/256/0/180] [15787.084407] LNet: Added LNI 192.168.202.136@tcp101 [8/256/0/180] [15788.196665] LNet: Added LNI 192.168.202.136@tcp102 [8/256/0/180] [15789.457547] LNet: Added LNI 192.168.202.136@tcp103 [8/256/0/180] [15790.668675] LNet: Added LNI 192.168.202.136@tcp104 [8/256/0/180] [15791.810132] LNet: Added LNI 192.168.202.136@tcp105 [8/256/0/180] [15793.070862] LNet: Added LNI 192.168.202.136@tcp106 [8/256/0/180] [15794.103575] LNet: Added LNI 192.168.202.136@tcp107 [8/256/0/180] [15795.475070] LNet: Added LNI 192.168.202.136@tcp108 [8/256/0/180] [15796.552428] LNet: Added LNI 192.168.202.136@tcp109 [8/256/0/180] [15797.814386] LNet: Added LNI 192.168.202.136@tcp110 [8/256/0/180] [15798.838719] LNet: Added LNI 192.168.202.136@tcp111 [8/256/0/180] [15799.938465] LNet: Added LNI 192.168.202.136@tcp112 [8/256/0/180] [15801.088343] LNet: Added LNI 192.168.202.136@tcp113 [8/256/0/180] [15801.956827] LNet: Added LNI 192.168.202.136@tcp114 [8/256/0/180] [15802.946050] LNet: Added LNI 192.168.202.136@tcp115 [8/256/0/180] [15804.232769] LNet: Added LNI 192.168.202.136@tcp116 [8/256/0/180] [15804.974686] LNet: Added LNI 192.168.202.136@tcp117 [8/256/0/180] [15806.250661] LNet: Added LNI 192.168.202.136@tcp118 [8/256/0/180] [15807.205641] LNet: Added LNI 192.168.202.136@tcp119 [8/256/0/180] [15807.933535] LNet: Added LNI 192.168.202.136@tcp120 [8/256/0/180] [15808.893863] LNet: Added LNI 192.168.202.136@tcp121 [8/256/0/180] [15810.392901] LNet: Added LNI 192.168.202.136@tcp122 [8/256/0/180] [15811.839331] LNet: Added LNI 192.168.202.136@tcp123 [8/256/0/180] [15813.504598] LNet: Added LNI 192.168.202.136@tcp124 [8/256/0/180] [15814.758624] LNet: Added LNI 192.168.202.136@tcp125 [8/256/0/180] [15816.570899] LNet: Added LNI 192.168.202.136@tcp126 [8/256/0/180] [15817.794153] LNet: Added LNI 192.168.202.136@tcp127 [8/256/0/180] [15819.111862] LNet: Added LNI 192.168.202.136@tcp128 [8/256/0/180] [15820.206494] LNet: Added LNI 192.168.202.136@tcp129 [8/256/0/180] [15821.318851] LNet: Added LNI 192.168.202.136@tcp130 [8/256/0/180] [15822.365524] LNet: Added LNI 192.168.202.136@tcp131 [8/256/0/180] [15823.850699] LNet: Added LNI 192.168.202.136@tcp132 [8/256/0/180] [15825.026657] LNet: Added LNI 192.168.202.136@tcp133 [8/256/0/180] [15826.111972] LNet: Added LNI 192.168.202.136@tcp134 [8/256/0/180] [15827.190255] LNet: Added LNI 192.168.202.136@tcp135 [8/256/0/180] [15836.049172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15836.117879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15837.258975] Lustre: 464786:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15837.384507] LNetError: 464815:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15837.403617] LNetError: Protocol error connecting to 192.168.202.136@tcp117 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15837.423529] LNetError: Skipped 1 previous similar message [15837.522793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15837.556660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15837.601724] LNet: 5 local NIs in recovery (showing 5): 192.168.202.136@tcp125, 192.168.202.136@tcp126, 192.168.202.136@tcp128, 192.168.202.136@tcp127, 192.168.202.136@tcp130 [15837.656523] Lustre: lustre-MDT0000: new disk, initializing [15837.769464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15837.789558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15838.496608] LNetError: 462241:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15838.511621] LNetError: 462241:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [15838.526685] LNetError: Protocol error connecting to 192.168.202.136@tcp104 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15838.550277] LNetError: Skipped 9 previous similar messages [15840.672600] LNetError: 464813:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15840.696074] LNetError: 464813:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15840.709944] LNetError: Protocol error connecting to 192.168.202.136@tcp125 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15840.714931] LNetError: Skipped 7 previous similar messages [15843.907688] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15844.773577] LNetError: 464808:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15844.790633] LNetError: 464808:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15844.802980] LNetError: Protocol error connecting to 192.168.202.136@tcp125 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15844.820990] LNetError: Skipped 1 previous similar message [15852.962895] LNetError: 464806:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15852.985532] LNetError: Protocol error connecting to 192.168.202.136@tcp125 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15859.430911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15859.593820] Lustre: 464804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15859.656871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15859.660868] Lustre: Skipped 1 previous similar message [15859.784558] Lustre: lustre-MDT0001: new disk, initializing [15859.898586] Lustre: 465610:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15859.918727] Lustre: 465610:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15860.095302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15860.154111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15860.178731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15862.211731] LNetError: 464821:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15862.231309] LNetError: 464821:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15862.241223] LNetError: Protocol error connecting to 192.168.202.136@tcp134 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15862.251452] LNetError: Skipped 6 previous similar messages [15866.952435] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15869.535712] Lustre: 465959:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15869.548382] Lustre: 465959:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15879.458774] LNetError: 464817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15879.487117] LNetError: 464817:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15879.495380] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15879.511173] LNetError: Protocol error connecting to 192.168.202.136@tcp117 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15879.511188] LNetError: Skipped 9 previous similar messages [15887.200596] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15895.334324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15895.708099] Lustre: lustre-OST0000: new disk, initializing [15895.715133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15895.725373] Lustre: 466783:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15895.859772] Lustre: 466788:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15895.872165] Lustre: 466788:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15895.974530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15897.055163] LNet: 2 local NIs in recovery (showing 2): 192.168.202.136@tcp134, 192.168.202.136@tcp125 [15904.303688] Lustre: 467097:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15904.332537] Lustre: 467097:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15904.433160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15904.444485] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15904.672675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15904.838605] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15913.252614] LNetError: 462243:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15913.260884] LNetError: 462243:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15913.264720] LNetError: Protocol error connecting to 192.168.202.136@tcp124 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15913.281805] LNetError: Skipped 18 previous similar messages [15916.504274] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15925.228050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15925.228594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15925.239100] LustreError: Skipped 1 previous similar message [15925.262252] Lustre: Skipped 1 previous similar message [15925.266807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15930.341514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15930.346212] Lustre: Skipped 2 previous similar messages [15930.900264] Lustre: server umount lustre-OST0000 complete [15935.457161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15935.473786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15935.488074] Lustre: Skipped 1 previous similar message [15940.588586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15940.601085] Lustre: Skipped 1 previous similar message [15941.823662] Lustre: server umount lustre-MDT0000 complete [15947.283791] LustreError: 464794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151313 with bad export cookie 1779918611197515084 [15947.289367] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15947.290748] LustreError: 464794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15947.801906] Lustre: server umount lustre-MDT0001 complete [15960.574480] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15964.125251] Key type lgssc unregistered [15964.817623] LNet: 468699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15964.848086] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15965.919344] LNet: Removed LNI 192.168.202.136@tcp [15965.941665] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15965.960369] LNet: Removed LNI 192.168.202.136@tcp100 [15967.011087] LNet: Removed LNI 192.168.202.136@tcp101 [15967.014835] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15967.033366] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15967.065332] LNet: Removed LNI 192.168.202.136@tcp102 [15968.095248] LNet: Removed LNI 192.168.202.136@tcp103 [15969.119192] LNet: Removed LNI 192.168.202.136@tcp104 [15969.123593] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15969.143906] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15969.150879] LNet: Removed LNI 192.168.202.136@tcp105 [15970.207274] LNet: Removed LNI 192.168.202.136@tcp106 [15971.233816] LNet: Removed LNI 192.168.202.136@tcp107 [15972.255212] LNet: Removed LNI 192.168.202.136@tcp108 [15973.280312] LNet: Removed LNI 192.168.202.136@tcp109 [15973.285865] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15973.305031] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15973.319430] LNet: Removed LNI 192.168.202.136@tcp110 [15974.367334] LNet: Removed LNI 192.168.202.136@tcp111 [15975.391480] LNet: Removed LNI 192.168.202.136@tcp112 [15976.416920] LNet: Removed LNI 192.168.202.136@tcp113 [15977.439598] LNet: Removed LNI 192.168.202.136@tcp114 [15978.463260] LNet: Removed LNI 192.168.202.136@tcp115 [15979.487661] LNet: Removed LNI 192.168.202.136@tcp116 [15980.511751] LNet: Removed LNI 192.168.202.136@tcp117 [15981.535209] LNet: Removed LNI 192.168.202.136@tcp118 [15981.539410] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15981.556770] LNetError: 468699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15981.563596] LNet: Removed LNI 192.168.202.136@tcp119 [15982.624609] LNet: Removed LNI 192.168.202.136@tcp120 [15983.648889] LNet: Removed LNI 192.168.202.136@tcp121 [15984.671259] LNet: Removed LNI 192.168.202.136@tcp122 [15985.695953] LNet: Removed LNI 192.168.202.136@tcp123 [15986.719192] LNet: Removed LNI 192.168.202.136@tcp124 [15986.733780] LNet: Removed LNI 192.168.202.136@tcp125 [15987.743615] LNet: Removed LNI 192.168.202.136@tcp126 [15988.767237] LNet: Removed LNI 192.168.202.136@tcp127 [15989.791264] LNet: Removed LNI 192.168.202.136@tcp128 [15990.817388] LNet: Removed LNI 192.168.202.136@tcp129 [15991.840660] LNet: Removed LNI 192.168.202.136@tcp130 [15992.863204] LNet: Removed LNI 192.168.202.136@tcp131 [15993.889096] LNet: Removed LNI 192.168.202.136@tcp132 [15994.911201] LNet: Removed LNI 192.168.202.136@tcp133 [15994.914508] LNet: Removed LNI 192.168.202.136@tcp134 [15995.945510] LNet: Removed LNI 192.168.202.136@tcp135 [15997.145473] Key type .llcrypt unregistered [15997.150367] Key type ._llcrypt unregistered [16024.043462] Key type ._llcrypt registered [16024.048040] Key type .llcrypt registered [16024.236073] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16040.323226] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16041.650283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16041.802239] alg: No test for adler32 (adler32-zlib) [16042.989965] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [16043.395923] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16045.063474] Key type lgssc registered [16046.453859] Lustre: Echo OBD driver; http://www.lustre.org/ [16095.286444] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:30:58 (1789151458) [16118.115331] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16121.294383] Key type lgssc unregistered [16121.654984] LNet: 472796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16121.662576] LNetError: 472796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16121.680135] LNet: Removed LNI 192.168.202.136@tcp [16122.536433] Key type .llcrypt unregistered [16122.543149] Key type ._llcrypt unregistered [16141.142436] Key type ._llcrypt registered [16141.144690] Key type .llcrypt registered [16141.236135] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16142.314614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16142.555223] alg: No test for adler32 (adler32-zlib) [16143.624766] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [16143.812559] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16145.447882] Key type lgssc registered [16146.902468] Lustre: Echo OBD driver; http://www.lustre.org/ [16158.013386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16158.075034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16159.467397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16159.539682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16159.638900] Lustre: lustre-MDT0000: new disk, initializing [16159.739658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16159.764920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16165.983591] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16177.887050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16178.057357] Lustre: 474007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16178.123936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16178.129424] Lustre: Skipped 1 previous similar message [16178.193297] Lustre: lustre-MDT0001: new disk, initializing [16178.288556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16178.333370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16178.343383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16184.312576] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16194.698696] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16201.481655] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16208.976800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16209.450785] Lustre: lustre-OST0000: new disk, initializing [16209.455184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16209.462967] Lustre: 475979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16209.573809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16215.617610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16215.631808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16215.800055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16217.690422] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16229.219676] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16234.773228] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing lnet_if_list [16241.962603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16241.993054] LNet: Added LNI 192.168.202.136@tcp10 [8/256/0/180] [16248.758329] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16248.788655] LNet: Added LNI 192.168.202.136@tcp11 [8/256/0/180] [16254.155867] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16254.210934] LNet: Added LNI 192.168.202.136@tcp12 [8/256/0/180] [16259.715326] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16259.737611] LNet: Added LNI 192.168.202.136@tcp13 [8/256/0/180] [16265.561466] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16265.639722] LNet: Added LNI 192.168.202.136@tcp14 [8/256/0/180] [16270.150593] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16270.175793] LNet: Added LNI 192.168.202.136@tcp15 [8/256/0/180] [16275.171460] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16275.188679] LNet: Added LNI 192.168.202.136@tcp16 [8/256/0/180] [16280.341425] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16280.377439] LNet: Added LNI 192.168.202.136@tcp17 [8/256/0/180] [16286.761030] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16286.809228] LNet: Added LNI 192.168.202.136@tcp18 [8/256/0/180] [16292.157722] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16292.195341] LNet: Added LNI 192.168.202.136@tcp19 [8/256/0/180] [16296.257559] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16296.278373] LNet: Added LNI 192.168.202.136@tcp20 [8/256/0/180] [16301.624373] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16301.652504] LNet: Added LNI 192.168.202.136@tcp21 [8/256/0/180] [16308.084281] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16308.163973] LNet: Added LNI 192.168.202.136@tcp22 [8/256/0/180] [16314.945103] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16314.995491] LNet: Added LNI 192.168.202.136@tcp23 [8/256/0/180] [16321.844961] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16321.890452] LNet: Added LNI 192.168.202.136@tcp24 [8/256/0/180] [16329.330463] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16329.440302] LNet: Added LNI 192.168.202.136@tcp25 [8/256/0/180] [16336.425160] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16336.446506] LNet: Added LNI 192.168.202.136@tcp26 [8/256/0/180] [16342.609076] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16342.636919] LNet: Added LNI 192.168.202.136@tcp27 [8/256/0/180] [16349.527548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16349.553266] LNet: Added LNI 192.168.202.136@tcp28 [8/256/0/180] [16357.366246] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16357.389158] LNet: Added LNI 192.168.202.136@tcp29 [8/256/0/180] [16364.515464] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16364.547360] LNet: Added LNI 192.168.202.136@tcp30 [8/256/0/180] [16371.108552] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16371.148422] LNet: Added LNI 192.168.202.136@tcp31 [8/256/0/180] [16378.020961] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16378.064438] LNet: Added LNI 192.168.202.136@tcp32 [8/256/0/180] [16385.499061] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16385.531963] LNet: Added LNI 192.168.202.136@tcp33 [8/256/0/180] [16392.295784] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16392.323176] LNet: Added LNI 192.168.202.136@tcp34 [8/256/0/180] [16399.983828] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16400.040754] LNet: Added LNI 192.168.202.136@tcp35 [8/256/0/180] [16407.882217] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16407.935043] LNet: Added LNI 192.168.202.136@tcp36 [8/256/0/180] [16415.982992] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16416.014982] LNet: Added LNI 192.168.202.136@tcp37 [8/256/0/180] [16425.018655] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16425.065265] LNet: Added LNI 192.168.202.136@tcp38 [8/256/0/180] [16433.279490] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16433.342738] LNet: Added LNI 192.168.202.136@tcp39 [8/256/0/180] [16441.248337] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16441.262518] LNet: Added LNI 192.168.202.136@tcp40 [8/256/0/180] [16454.113325] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16454.179878] LNet: Added LNI 192.168.202.136@tcp41 [8/256/0/180] [16469.941185] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16469.978975] LNet: Added LNI 192.168.202.136@tcp42 [8/256/0/180] [16480.580369] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16480.607268] LNet: Added LNI 192.168.202.136@tcp43 [8/256/0/180] [16491.090978] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16491.143392] LNet: Added LNI 192.168.202.136@tcp44 [8/256/0/180] [16501.712538] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16501.735815] LNet: Added LNI 192.168.202.136@tcp45 [8/256/0/180] [16510.539843] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16510.587844] LNet: Added LNI 192.168.202.136@tcp46 [8/256/0/180] [16520.838339] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16520.853595] LNet: Added LNI 192.168.202.136@tcp47 [8/256/0/180] [16529.184162] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16529.213178] LNet: Added LNI 192.168.202.136@tcp48 [8/256/0/180] [16537.788976] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16537.820516] LNet: Added LNI 192.168.202.136@tcp49 [8/256/0/180] [16546.498880] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16546.539382] LNet: Added LNI 192.168.202.136@tcp50 [8/256/0/180] [16567.264630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16567.276983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16567.294972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16569.314967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16569.318564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16569.329657] Lustre: Skipped 1 previous similar message [16569.384177] Lustre: Skipped 1 previous similar message [16570.851695] Lustre: server umount lustre-MDT0000 complete [16575.096647] LustreError: 474000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789151941 with bad export cookie 12189283362008837340 [16575.098333] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16575.108461] LustreError: 474000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16575.535680] Lustre: server umount lustre-MDT0001 complete [16594.337679] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16595.940649] Lustre: 473281:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789151945/real 1789151945] req@00000000a9be2f1e x1876061332883200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789151961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16595.984609] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16596.960357] Lustre: 473280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789151946/real 1789151946] req@00000000baf155e4 x1876061332883328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789151962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16600.097538] Lustre: 473280:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789151950/real 1789151950] req@00000000615907cd x1876061332883456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789151966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16604.947646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16606.175169] Lustre: 473282:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789151956/real 1789151956] req@00000000ee935c46 x1876061332883840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789151972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16606.207664] Lustre: 473282:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16611.295408] Lustre: 473279:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789151961/real 1789151961] req@0000000074de8577 x1876061332883968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789151977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16620.642692] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa92908a2db1db6d0 [16620.651601] Lustre: MGC192.168.202.136@tcp: Connection restored to 0@lo (at 0@lo) [16620.862288] LustreError: 475972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16620.876893] LustreError: 475972:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [16620.918278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16621.924379] LustreError: 500226:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16621.961178] LustreError: 500226:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16624.995729] LustreError: 475973:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16625.005736] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16625.013270] LustreError: 475973:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16626.400370] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16630.245892] LustreError: 500228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16630.263393] LustreError: 500228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16635.397755] LustreError: 500228:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16635.428330] LustreError: 500228:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16639.053667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16640.032516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16644.874149] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16645.135286] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16654.815646] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16662.004559] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16680.930783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16680.934392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16680.945465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16680.949084] Lustre: Skipped 1 previous similar message [16686.055789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16686.075489] Lustre: Skipped 2 previous similar messages [16691.180478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16691.199674] Lustre: Skipped 1 previous similar message [16695.263436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16695.410479] Lustre: server umount lustre-OST0000 complete [16701.409463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16701.414503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16701.418201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16701.432513] Lustre: Skipped 2 previous similar messages [16701.440419] Lustre: Skipped 1 previous similar message [16706.530818] LustreError: 500231:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16706.549525] LustreError: 500231:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16708.449751] Lustre: server umount lustre-MDT0000 complete [16713.005484] LustreError: 474001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152079 with bad export cookie 12189283362008839888 [16713.013316] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16713.022127] LustreError: 474001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16713.541643] Lustre: server umount lustre-MDT0001 complete [16729.181798] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16732.803507] Key type lgssc unregistered [16733.229846] LNet: 503014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16733.243722] LNetError: 503014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16733.254644] LNet: Removed LNI 192.168.202.136@tcp [16733.257682] LNet: Removed LNI 192.168.202.136@tcp10 [16733.259807] LNet: Removed LNI 192.168.202.136@tcp11 [16733.262956] LNet: Removed LNI 192.168.202.136@tcp12 [16733.266427] LNet: Removed LNI 192.168.202.136@tcp13 [16733.273496] LNet: Removed LNI 192.168.202.136@tcp14 [16733.278357] LNet: Removed LNI 192.168.202.136@tcp15 [16733.286701] LNet: Removed LNI 192.168.202.136@tcp16 [16733.295825] LNet: Removed LNI 192.168.202.136@tcp17 [16733.300619] LNet: Removed LNI 192.168.202.136@tcp18 [16733.304559] LNet: Removed LNI 192.168.202.136@tcp19 [16733.311580] LNet: Removed LNI 192.168.202.136@tcp20 [16733.316477] LNet: Removed LNI 192.168.202.136@tcp21 [16733.320731] LNet: Removed LNI 192.168.202.136@tcp22 [16733.328537] LNet: Removed LNI 192.168.202.136@tcp23 [16733.341496] LNet: Removed LNI 192.168.202.136@tcp24 [16733.356136] LNet: Removed LNI 192.168.202.136@tcp25 [16733.364732] LNet: Removed LNI 192.168.202.136@tcp26 [16733.374630] LNet: Removed LNI 192.168.202.136@tcp27 [16733.379512] LNet: Removed LNI 192.168.202.136@tcp28 [16733.383035] LNet: Removed LNI 192.168.202.136@tcp29 [16733.389336] LNet: Removed LNI 192.168.202.136@tcp30 [16733.392581] LNet: Removed LNI 192.168.202.136@tcp31 [16733.396659] LNet: Removed LNI 192.168.202.136@tcp32 [16733.401217] LNet: Removed LNI 192.168.202.136@tcp33 [16733.405933] LNet: Removed LNI 192.168.202.136@tcp34 [16733.408695] LNet: Removed LNI 192.168.202.136@tcp35 [16733.413312] LNet: Removed LNI 192.168.202.136@tcp36 [16733.416505] LNet: Removed LNI 192.168.202.136@tcp37 [16733.420358] LNet: Removed LNI 192.168.202.136@tcp38 [16733.423864] LNet: Removed LNI 192.168.202.136@tcp39 [16733.428307] LNet: Removed LNI 192.168.202.136@tcp40 [16733.432258] LNet: Removed LNI 192.168.202.136@tcp41 [16733.434995] LNet: Removed LNI 192.168.202.136@tcp42 [16733.442396] LNet: Removed LNI 192.168.202.136@tcp43 [16733.446127] LNet: Removed LNI 192.168.202.136@tcp44 [16733.453570] LNet: Removed LNI 192.168.202.136@tcp45 [16733.467222] LNet: Removed LNI 192.168.202.136@tcp46 [16733.471529] LNet: Removed LNI 192.168.202.136@tcp47 [16733.475424] LNet: Removed LNI 192.168.202.136@tcp48 [16733.477933] LNet: Removed LNI 192.168.202.136@tcp49 [16733.492272] LNet: Removed LNI 192.168.202.136@tcp50 [16734.537824] Key type .llcrypt unregistered [16734.539387] Key type ._llcrypt unregistered [16759.874221] Key type ._llcrypt registered [16759.880930] Key type .llcrypt registered [16760.040969] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16778.544260] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16779.992928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16780.086263] alg: No test for adler32 (adler32-zlib) [16781.354630] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [16781.809681] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16783.575364] Key type lgssc registered [16784.457666] Lustre: Echo OBD driver; http://www.lustre.org/ [16821.761869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16821.785768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16823.124358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16823.198094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16823.421798] Lustre: lustre-MDT0000: new disk, initializing [16823.536243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16823.554703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16828.777572] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16840.415742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16840.595168] Lustre: 506214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16840.692825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16840.698338] Lustre: Skipped 1 previous similar message [16840.844665] Lustre: lustre-MDT0001: new disk, initializing [16841.090595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16841.145044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16841.163559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16847.073877] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16858.550777] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16866.552919] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16878.230056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16878.577295] Lustre: *** cfs_fail_loc=90f, val=0*** [16878.621082] Lustre: lustre-OST0000: new disk, initializing [16878.623758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16878.629403] Lustre: 508235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16878.750233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16884.417828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16884.443209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16884.740538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16889.661111] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16901.350291] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16910.308313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16910.309462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16910.319679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16910.329333] LustreError: Skipped 1 previous similar message [16915.429787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16915.440305] Lustre: Skipped 2 previous similar messages [16916.506742] Lustre: server umount lustre-OST0000 complete [16923.105347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16923.110979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16923.117554] Lustre: Skipped 1 previous similar message [16923.121507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16925.666878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16925.667583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16925.678016] Lustre: Skipped 1 previous similar message [16927.010682] Lustre: server umount lustre-MDT0000 complete [16931.047895] LustreError: 506208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152297 with bad export cookie 14674144439096099683 [16931.060584] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16931.060713] LustreError: 506208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16931.495611] Lustre: server umount lustre-MDT0001 complete [16945.703321] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16949.687916] Key type lgssc unregistered [16950.158814] LNet: 510145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16950.169041] LNetError: 510145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16950.207609] LNet: Removed LNI 192.168.202.136@tcp [16951.225230] Key type .llcrypt unregistered [16951.234582] Key type ._llcrypt unregistered [16975.948946] Key type ._llcrypt registered [16975.951832] Key type .llcrypt registered [16976.124484] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16977.513991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16977.770181] alg: No test for adler32 (adler32-zlib) [16978.895636] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [16979.305207] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16981.071293] Key type lgssc registered [16982.363393] Lustre: Echo OBD driver; http://www.lustre.org/ [16991.336785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16991.356707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16992.844888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16997.600717] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17002.990139] LustreError: 511415:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17003.023539] LustreError: 511415:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [17007.547084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17008.068986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17014.444077] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17023.558464] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17030.567477] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17039.605681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17039.992418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17049.974922] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17061.838223] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17073.207980] loop0: detected capacity change from 0 to 204800000 [17073.525659] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17073.532325] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17073.564847] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17081.502716] loop0: detected capacity change from 0 to 204800000 [17081.957272] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17081.986657] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17082.000762] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [17088.898687] loop0: detected capacity change from 0 to 204800000 [17095.397708] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17095.736218] Lustre: fs15246-OST0000: new disk, initializing [17095.742640] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [17095.749557] Lustre: 515010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17095.952820] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17103.787427] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17114.552309] loop1: detected capacity change from 0 to 204800000 [17121.076420] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17121.202377] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [17121.271884] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [17121.445981] Lustre: fs15246-MDT0000: new disk, initializing [17121.713688] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17121.740284] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17121.882326] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [17122.913204] Lustre: 516133:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f3d007c3 x1876062209365504/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789152504 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [17127.841972] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17127.997289] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17128.019334] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17128.062885] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17152.995482] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17153.034536] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17158.121514] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [17158.348407] Lustre: server umount fs15246-MDT0000 complete [17163.252690] LustreError: 513671:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17168.361057] LustreError: 511410:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17171.385527] Lustre: server umount fs15246-OST0000 complete [17190.880867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17190.888558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17190.907644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17191.913993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17191.930269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17193.953143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17193.956674] Lustre: Skipped 2 previous similar messages [17199.080061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17199.086164] Lustre: Skipped 1 previous similar message [17204.195493] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17204.559850] Lustre: server umount lustre-OST0000 complete [17212.897313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17212.915217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17212.934102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17212.937936] Lustre: Skipped 2 previous similar messages [17216.012122] Lustre: server umount lustre-MDT0000 complete [17219.575455] LustreError: 511411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17219.599995] LustreError: 511411:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [17220.743175] LustreError: 517272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152586 with bad export cookie 1933874913264329055 [17220.757269] LustreError: 517272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17220.760337] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17221.194925] Lustre: server umount lustre-MDT0001 complete [17234.160224] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [17237.604900] Key type lgssc unregistered [17237.996881] LNet: 519238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17238.009313] LNetError: 519238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17238.028405] LNet: Removed LNI 192.168.202.136@tcp [17239.146453] Key type .llcrypt unregistered [17239.151422] Key type ._llcrypt unregistered [17261.934408] Key type ._llcrypt registered [17261.939239] Key type .llcrypt registered [17262.072907] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [17262.985988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17263.029749] alg: No test for adler32 (adler32-zlib) [17264.160089] Lustre: Lustre: Build Version: 2.17.57_131_gc13dec9 [17264.360643] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [17266.095424] Key type lgssc registered [17267.651959] Lustre: Echo OBD driver; http://www.lustre.org/ [17317.523431] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [17329.222997] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [17376.024564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17376.087061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17377.712205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17377.776527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17377.934393] Lustre: lustre-MDT0000: new disk, initializing [17378.066256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17378.093964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17387.706603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17402.512805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17402.758594] Lustre: 523713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17402.962355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17402.983420] Lustre: Skipped 1 previous similar message [17403.109077] Lustre: lustre-MDT0001: new disk, initializing [17403.251126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17403.330908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17403.350348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17408.218691] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17420.239123] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17428.524501] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17437.749688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17438.234669] Lustre: lustre-OST0000: new disk, initializing [17438.239957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17438.246835] Lustre: 525679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17438.415403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17444.577177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17444.610362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17444.809785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17446.920454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [17459.922465] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17466.309219] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17466.655465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17472.520037] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17472.746957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17475.558291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17475.564231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17475.586135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17475.626574] Lustre: Skipped 1 previous similar message [17480.685084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17480.694711] Lustre: Skipped 1 previous similar message [17485.797496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17485.801457] Lustre: Skipped 2 previous similar messages [17489.375405] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17489.684470] Lustre: server umount lustre-OST0000 complete [17495.535797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17495.547459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17495.567454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17501.153032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17501.168522] Lustre: Skipped 3 previous similar messages [17509.343150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17509.748627] Lustre: server umount lustre-MDT0000 complete [17515.542103] LustreError: 523705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789152881 with bad export cookie 1567320915727554127 [17515.563195] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17515.568066] LustreError: 523705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17516.086240] Lustre: server umount lustre-MDT0001 complete [17549.725548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [17568.788391] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [17613.112767] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 17288 sec ========= 14:56:17 (1789152977) [17615.498716] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:56:20 (1789152980) === [17620.580781] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:56:24 (1789152984) === [17672.153087] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [17676.376783] Key type lgssc unregistered [17676.742195] LNet: 532362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17676.754863] LNetError: 532362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17677.800791] LNet: Removed LNI 192.168.202.136@tcp [17678.807553] Key type .llcrypt unregistered [17678.813466] Key type ._llcrypt unregistered