[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500980899 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003181] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010053] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.020292] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024278] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.030402] *** VALIDATE proc *** [ 0.032234] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035220] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.041211] debug: unmapping init [mem 0xffffffff9da59000-0xffffffff9da60fff] [ 0.043185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044642] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045018] ... version: 2 [ 0.046007] ... bit width: 48 [ 0.046974] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051264] rcu: Hierarchical SRCU implementation. [ 0.053370] smp: Bringing up secondary CPUs ... [ 0.054581] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.058234] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141000] node 0 deferred pages initialised in 78ms [ 0.145157] devtmpfs: initialized [ 0.146239] x86/mm: Memory block size: 128MB [ 0.148587] gcov: version magic: 0x41383552 [ 0.152310] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154171] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156309] pinctrl core: initialized pinctrl subsystem [ 0.158216] [ 0.158764] ************************************************************* [ 0.161012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ** ** [ 0.165011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167012] ** ** [ 0.170014] ** This means that this kernel is built to expose internal ** [ 0.172011] ** IOMMU data structures, which may compromise security on ** [ 0.175015] ** your system. ** [ 0.177012] ** ** [ 0.179012] ** If you see this message and you are not debugging the ** [ 0.181012] ** kernel, report this immediately to your vendor! ** [ 0.184019] ** ** [ 0.186009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188010] ************************************************************* [ 0.190776] NET: Registered protocol family 16 [ 0.193444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203030] cpuidle: using governor menu [ 0.204966] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.207472] PCI: Using configuration type 1 for base access [ 0.210148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218068] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225034] cryptd: max_cpu_qlen set to 1000 [ 0.227237] ACPI: Added _OSI(Module Device) [ 0.229015] ACPI: Added _OSI(Processor Device) [ 0.231012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.237000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245008] ACPI: Interpreter enabled [ 0.246054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248006] ACPI: Using IOAPIC for interrupt routing [ 0.250124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253452] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263540] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276385] acpiphp: Slot [2] registered [ 0.278110] acpiphp: Slot [5] registered [ 0.279098] acpiphp: Slot [6] registered [ 0.281084] acpiphp: Slot [7] registered [ 0.282079] acpiphp: Slot [8] registered [ 0.283081] acpiphp: Slot [9] registered [ 0.284083] acpiphp: Slot [10] registered [ 0.286128] acpiphp: Slot [3] registered [ 0.287080] acpiphp: Slot [4] registered [ 0.288110] acpiphp: Slot [11] registered [ 0.290065] acpiphp: Slot [12] registered [ 0.291101] acpiphp: Slot [13] registered [ 0.292108] acpiphp: Slot [14] registered [ 0.294114] acpiphp: Slot [15] registered [ 0.295073] acpiphp: Slot [16] registered [ 0.297081] acpiphp: Slot [17] registered [ 0.298081] acpiphp: Slot [18] registered [ 0.299076] acpiphp: Slot [19] registered [ 0.300068] acpiphp: Slot [20] registered [ 0.301075] acpiphp: Slot [21] registered [ 0.302122] acpiphp: Slot [22] registered [ 0.303072] acpiphp: Slot [23] registered [ 0.304081] acpiphp: Slot [24] registered [ 0.306079] acpiphp: Slot [25] registered [ 0.307093] acpiphp: Slot [26] registered [ 0.309098] acpiphp: Slot [27] registered [ 0.311113] acpiphp: Slot [28] registered [ 0.312077] acpiphp: Slot [29] registered [ 0.314092] acpiphp: Slot [30] registered [ 0.316094] acpiphp: Slot [31] registered [ 0.318083] PCI host bridge to bus 0000:00 [ 0.319021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.332022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.334163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.336872] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.340471] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350824] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367884] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373775] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391752] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404373] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.440895] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.458013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482652] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.535206] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.545014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.553014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.575017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.587836] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.593013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.598014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.617015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.627399] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.634012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.652014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.662988] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.664329] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.666342] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.668922] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.671228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676029] iommu: Default domain type: Passthrough [ 0.678368] SCSI subsystem initialized [ 0.679126] ACPI: bus type USB registered [ 0.680076] usbcore: registered new interface driver usbfs [ 0.682045] usbcore: registered new interface driver hub [ 0.683054] usbcore: registered new device driver usb [ 0.685186] pps_core: LinuxPPS API ver. 1 registered [ 0.687011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.690056] PTP clock support registered [ 0.693054] EDAC MC: Ver: 3.0.0 [ 0.694110] PCI: Using ACPI for IRQ routing [ 0.695624] NetLabel: Initializing [ 0.697008] NetLabel: domain hash size = 128 [ 0.698009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.701082] NetLabel: unlabeled traffic allowed by default [ 0.703051] vgaarb: loaded [ 0.704248] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.705012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714801] clocksource: Switched to clocksource kvm-clock [ 0.820647] VFS: Disk quotas dquot_6.6.0 [ 0.822546] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825477] *** VALIDATE ramfs *** [ 0.826908] *** VALIDATE hugetlbfs *** [ 0.828660] pnp: PnP ACPI init [ 0.831387] pnp: PnP ACPI: found 6 devices [ 0.852063] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855394] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857505] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859069] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860567] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862670] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.865387] NET: Registered protocol family 2 [ 0.867880] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.872595] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.875870] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.881131] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.884519] TCP: Hash tables configured (established 65536 bind 65536) [ 0.887266] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.889711] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.892330] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895484] NET: Registered protocol family 1 [ 0.898136] RPC: Registered named UNIX socket transport module. [ 0.900113] RPC: Registered udp transport module. [ 0.901902] RPC: Registered tcp transport module. [ 0.903779] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.905967] NET: Registered protocol family 44 [ 0.908406] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.910869] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.913238] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.915733] PCI: CLS 0 bytes, default 64 [ 0.917510] Unpacking initramfs... [ 2.290393] debug: unmapping init [mem 0xffff96b7bcc54000-0xffff96b7bffbffff] [ 2.294125] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.295771] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.299385] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.777813] Initialise system trusted keyrings [ 2.779290] Key type blacklist registered [ 2.780967] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.796257] zbud: loaded [ 2.800508] *** VALIDATE nfs *** [ 2.802804] *** VALIDATE nfs4 *** [ 2.805193] pstore: using deflate compression [ 2.809415] Platform Keyring initialized [ 2.907049] NET: Registered protocol family 38 [ 2.908727] Key type asymmetric registered [ 2.910650] Asymmetric key parser 'x509' registered [ 2.912753] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.915934] io scheduler mq-deadline registered [ 2.917507] io scheduler kyber registered [ 2.918986] io scheduler bfq registered [ 2.921075] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.923835] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.927120] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.930288] ACPI: Power Button [PWRF] [ 2.935213] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.941328] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.956661] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.963712] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.976754] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.003670] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.030853] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.034639] Non-volatile memory driver v1.3 [ 3.035922] Linux agpgart interface v0.103 [ 3.061559] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.063818] vda: detected capacity change from 0 to 74829824 [ 3.075209] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.077348] vdb: detected capacity change from 0 to 1073741824 [ 3.088360] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.090598] vdc: detected capacity change from 0 to 2621440000 [ 3.108705] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.112753] vdd: detected capacity change from 0 to 2621440000 [ 3.133789] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.136126] vde: detected capacity change from 0 to 4294967296 [ 3.151272] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.153790] vdf: detected capacity change from 0 to 4294967296 [ 3.160216] libphy: Fixed MDIO Bus: probed [ 3.165268] usbcore: registered new interface driver usbserial_generic [ 3.167693] usbserial: USB Serial support registered for generic [ 3.169887] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.174567] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.176463] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.178730] mousedev: PS/2 mouse device common for all mice [ 3.181306] rtc_cmos 00:05: RTC can wake from S4 [ 3.183960] rtc_cmos 00:05: registered as rtc0 [ 3.186975] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.188115] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.195612] intel_pstate: CPU model not supported [ 3.200063] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.200300] hid: raw HID events driver (C) Jiri Kosina [ 3.207173] usbcore: registered new interface driver usbhid [ 3.208897] usbhid: USB HID core driver [ 3.210950] drop_monitor: Initializing network drop monitor service [ 3.211139] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.214686] Initializing XFRM netlink socket [ 3.219560] NET: Registered protocol family 10 [ 3.222572] Segment Routing with IPv6 [ 3.223835] NET: Registered protocol family 17 [ 3.225309] mpls_gso: MPLS GSO support [ 3.231294] RAS: Correctable Errors collector initialized. [ 3.234820] AVX version of gcm_enc/dec engaged. [ 3.237143] AES CTR mode by8 optimization enabled [ 3.301140] sched_clock: Marking stable (3301109356, 0)->(4182299612, -881190256) [ 3.305321] registered taskstats version 1 [ 3.307401] Loading compiled-in X.509 certificates [ 3.310082] zswap: loaded using pool lzo/zbud [ 3.330746] Key type big_key registered [ 3.339911] Key type encrypted registered [ 3.341659] ima: No TPM chip found, activating TPM-bypass! [ 3.343980] ima: Allocated hash algorithm: sha1 [ 3.345920] ima: No architecture policies found [ 3.347576] evm: Initialising EVM extended attributes: [ 3.349385] evm: security.selinux [ 3.350875] evm: security.ima [ 3.352158] evm: security.capability [ 3.353400] evm: HMAC attrs: 0x1 [ 3.355702] rtc_cmos 00:05: setting system clock to 2026-08-27 08:08:19 UTC (1787818099) [ 3.361404] debug: unmapping init [mem 0xffffffff9ea03000-0xffffffff9ebfffff] [ 3.363529] debug: unmapping init [mem 0xffffffff9d782000-0xffffffff9da58fff] [ 3.371566] Write protecting the kernel read-only data: 28672k [ 3.375791] debug: unmapping init [mem 0xffffffff9be03000-0xffffffff9bffffff] [ 3.379350] debug: unmapping init [mem 0xffffffff9c714000-0xffffffff9c7fffff] [ 3.408509] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.416308] systemd[1]: Detected virtualization kvm. [ 3.417874] systemd[1]: Detected architecture x86-64. [ 3.419669] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.444284] systemd[1]: No hostname configured. [ 3.445644] systemd[1]: Set hostname to . [ 3.447321] random: systemd: uninitialized urandom read (16 bytes read) [ 3.449237] systemd[1]: Initializing machine ID from random generator. [ 3.481667] random: ln: uninitialized urandom read (6 bytes read) [ 3.566127] random: systemd: uninitialized urandom read (16 bytes read) [ 3.568580] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.572995] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.576639] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.088575] device-mapper: uevent: version 1.0.3 [ 4.090743] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.698944] random: fast init done [ 4.717550] virtio_net virtio0 ens2: renamed from eth0 [ 4.728377] scsi host0: ata_piix [ 4.737225] scsi host1: ata_piix [ 4.738234] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.739986] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.376661] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.859045] random: crng init done [ 9.860620] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.135520] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.356338] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.639456] SELinux: Disabled at runtime. [ 11.711894] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.719852] systemd[1]: Detected virtualization kvm. [ 11.721788] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.245263] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.248820] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.253410] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.256574] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.260164] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.266348] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.272339] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ 12.369574] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.754490] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.070869] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.140170] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.254396] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.274893] EDAC sbridge: Ver: 1.1.2 [ 15.130297] Key type dns_resolver registered [ 15.456781] NFS: Registering the id_resolver key type [ 15.460091] Key type id_resolver registered [ 15.462610] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg648-server login: [ 41.487782] libcfs: loading out-of-tree module taints kernel. [ 41.507395] Key type ._llcrypt registered [ 41.509045] Key type .llcrypt registered [ 41.555252] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 50.500122] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 51.158181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.167535] alg: No test for adler32 (adler32-zlib) [ 52.197692] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 52.571294] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 54.200248] Key type lgssc registered [ 55.768400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.080238] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 81.347835] hrtimer: interrupt took 15821434 ns [ 113.469694] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 127.688298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.739679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.126876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.187729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.337798] Lustre: lustre-MDT0000: new disk, initializing [ 129.444518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.466174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 134.740404] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 148.206218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.362457] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.415191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.428906] Lustre: Skipped 1 previous similar message [ 148.624993] Lustre: lustre-MDT0001: new disk, initializing [ 148.737910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.831966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.846100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.146778] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.810736] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 166.762155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.047401] Lustre: lustre-OST0000: new disk, initializing [ 167.051232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.060735] Lustre: 8422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 167.153600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.029603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 169.039348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 169.181589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 173.250775] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.887146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 188.115129] Lustre: lustre-OST0001: new disk, initializing [ 188.119252] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.125741] Lustre: 9494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 188.186905] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.084831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 194.092985] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 194.158906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 194.680249] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.508035] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 212.465980] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 218.948925] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing check_logdir /tmp/testlogs/ [ 225.155298] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing yml_node [ 228.920223] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 231.441556] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 234.141954] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 235.931237] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 04:12:10 EDT 2026 [ 252.440747] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 253.658793] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 270.829545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.834453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.848039] Lustre: Skipped 1 previous similar message [ 275.388837] Lustre: server umount lustre-MDT0000 complete [ 279.856851] LustreError: 7897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818376 with bad export cookie 10693414396584996144 [ 279.861761] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 279.876892] LustreError: 7897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 281.056772] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.059266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.083794] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 281.106991] Lustre: Skipped 2 previous similar messages [ 281.111941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 281.115786] Lustre: Skipped 4 previous similar messages [ 284.068914] LustreError: 8854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.071665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 284.112904] Lustre: Skipped 1 previous similar message [ 289.248813] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.254714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 289.271777] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 289.298353] Lustre: Skipped 1 previous similar message [ 294.368336] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 294.613490] Lustre: server umount lustre-MDT0001 complete [ 299.047497] Lustre: server umount lustre-OST0000 complete [ 303.298677] Lustre: server umount lustre-OST0001 complete [ 309.869879] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 319.958372] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 357.559631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 357.827179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 357.856500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.933421] Lustre: lustre-MDT0000: new disk, initializing [ 358.001545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 358.021755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 362.684591] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 374.138721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.318183] Lustre: 15046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 374.384836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 374.392369] Lustre: Skipped 1 previous similar message [ 374.533649] Lustre: lustre-MDT0001: new disk, initializing [ 374.660279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 374.700413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 374.718232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 379.352356] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 389.457967] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 397.018509] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.505588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.830980] Lustre: lustre-OST0000: new disk, initializing [ 406.832927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.836102] Lustre: 17018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.911677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.210954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.234958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.468820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.850851] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 423.488790] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 428.881054] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.176950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 434.768803] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 435.053058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 439.266991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.275149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.276823] LustreError: Skipped 1 previous similar message [ 439.279855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.279862] Lustre: Skipped 2 previous similar messages [ 439.316675] Lustre: Skipped 2 previous similar messages [ 443.724599] Lustre: server umount lustre-OST0000 complete [ 449.519755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.535162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.549131] Lustre: Skipped 1 previous similar message [ 449.559941] Lustre: Skipped 1 previous similar message [ 454.322314] Lustre: server umount lustre-MDT0000 complete [ 454.633335] LustreError: 17596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.666925] LustreError: 17596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 458.423265] LustreError: 15038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818554 with bad export cookie 10693414396584997677 [ 458.442761] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.863913] Lustre: server umount lustre-MDT0001 complete [ 463.254106] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:15:58 (1787818558) [ 465.546675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 467.425062] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:16:02 (1787818562) [ 474.553237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.106335] LustreError: 19426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.176685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.226812] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 485.346911] LustreError: 19427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.365225] LustreError: 19427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 489.178415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.164926] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 502.135275] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.148095] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.448962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.265381] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 530.430504] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.673273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.685217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.706826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 541.714234] Lustre: Skipped 1 previous similar message [ 544.753551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.487289] Lustre: server umount lustre-OST0000 complete [ 551.395589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 551.410922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.427232] Lustre: Skipped 1 previous similar message [ 556.746593] Lustre: server umount lustre-MDT0000 complete [ 560.102470] LustreError: 19422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.120912] LustreError: 19422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 560.407733] LustreError: 19408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787818656 with bad export cookie 10693414396584998559 [ 560.414227] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.429321] LustreError: 19408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 560.878686] Lustre: server umount lustre-MDT0001 complete [ 572.667650] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 575.763221] Key type lgssc unregistered [ 576.201559] LNet: 22963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.208713] LNetError: 22963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.235060] LNet: Removed LNI 192.168.206.148@tcp [ 577.159156] Key type .llcrypt unregistered [ 577.162454] Key type ._llcrypt unregistered [ 599.929985] Key type ._llcrypt registered [ 599.933263] Key type .llcrypt registered [ 600.077972] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 601.549587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.693251] alg: No test for adler32 (adler32-zlib) [ 602.803478] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 603.067437] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 604.776146] Key type lgssc registered [ 605.749242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 614.262498] loop: module loaded [ 614.382815] loop0: detected capacity change from 0 to 204800000 [ 614.616299] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.625614] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.938798] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.610764] loop0: detected capacity change from 0 to 204800000 [ 620.788646] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.795367] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.800936] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.038930] loop0: detected capacity change from 0 to 204800000 [ 632.310327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.362052] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.742789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 633.887586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 634.075629] Lustre: lustre-MDT0000: new disk, initializing [ 634.224418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.256759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 639.176567] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 650.624191] loop1: detected capacity change from 0 to 204800000 [ 656.253889] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.488217] Lustre: lustre-OST07c6: new disk, initializing [ 656.492785] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 656.498638] Lustre: Skipped 1 previous similar message [ 656.502514] Lustre: 26147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 656.594304] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 662.137686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 662.148169] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 662.187358] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 662.888816] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 667.377169] Lustre: 25135:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 667.382161] Lustre: 25135:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 667.385396] Lustre: 25135:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 667.391068] Lustre: 25135:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 667.396882] Lustre: 25135:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 667.402715] Lustre: 25135:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 685.466702] Lustre: 26663:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 685.494869] Lustre: 26663:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.516111] Lustre: 26663:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.530250] Lustre: 26663:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 685.549036] Lustre: 26663:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 685.559700] Lustre: 26663:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.323626] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 690.332159] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 690.335717] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 690.340889] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.348130] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 690.352982] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.364709] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 690.372939] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.377266] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 690.382134] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.391877] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.398308] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.604858] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 693.614079] Lustre: 25134:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 693.621021] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 693.631854] Lustre: 25134:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.638981] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.647231] Lustre: 25134:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.653923] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 693.666377] Lustre: 25134:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.671984] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 693.681026] Lustre: 25134:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.687543] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.695745] Lustre: 25134:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.246893] Lustre: 26663:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 698.254237] Lustre: 26663:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.263512] Lustre: 26663:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 698.268038] Lustre: 26663:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.276783] Lustre: 26663:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.282588] Lustre: 26663:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.290620] Lustre: 26663:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 698.297144] Lustre: 26663:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.304767] Lustre: 26663:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 698.312308] Lustre: 26663:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.318386] Lustre: 26663:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.331083] Lustre: 26663:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.533312] Lustre: 26663:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 709.546950] Lustre: 26663:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 709.563467] Lustre: 26663:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 709.572479] Lustre: 26663:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 709.585614] Lustre: 26663:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.596485] Lustre: 26663:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 709.602923] Lustre: 26663:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 709.609857] Lustre: 26663:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 709.618521] Lustre: 26663:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 709.623151] Lustre: 26663:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 709.631906] Lustre: 26663:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.644991] Lustre: 26663:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 759.267397] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 759.275778] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 764.387704] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 764.930522] Lustre: server umount lustre-OST07c6 complete [ 775.426640] Lustre: server umount lustre-MDT0000 complete [ 792.768587] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 795.795844] Key type lgssc unregistered [ 796.191927] LNet: 28711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.202453] LNetError: 28711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.244208] LNet: Removed LNI 192.168.206.148@tcp [ 797.169715] Key type .llcrypt unregistered [ 797.172505] Key type ._llcrypt unregistered [ 814.280068] Key type ._llcrypt registered [ 814.282053] Key type .llcrypt registered [ 814.394989] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 815.244223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 815.302565] alg: No test for adler32 (adler32-zlib) [ 816.359736] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 816.544548] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 818.176182] Key type lgssc registered [ 819.111972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.340498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 825.358827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.724956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 830.676137] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 836.072817] LustreError: 29981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.092385] LustreError: 29981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 838.362828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.756228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 843.689469] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 851.128046] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 856.500533] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 862.176834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 862.566089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.076635] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 871.795565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 874.693536] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 887.266981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.272829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.279079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.289009] Lustre: Skipped 1 previous similar message [ 887.314376] Lustre: Skipped 1 previous similar message [ 892.395488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 892.407357] Lustre: Skipped 1 previous similar message [ 893.042639] Lustre: server umount lustre-OST0000 complete [ 897.506909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.519853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.327750] Lustre: server umount lustre-MDT0000 complete [ 902.626178] LustreError: 29981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.649791] LustreError: 29981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 905.416030] LustreError: 29961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819001 with bad export cookie 16064708380641550066 [ 905.428972] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 905.775345] Lustre: server umount lustre-MDT0001 complete [ 914.699467] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 917.250969] Key type lgssc unregistered [ 917.593655] LNet: 33719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.602193] LNetError: 33719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.631652] LNet: Removed LNI 192.168.206.148@tcp [ 918.470261] Key type .llcrypt unregistered [ 918.472398] Key type ._llcrypt unregistered [ 935.932244] Key type ._llcrypt registered [ 935.934062] Key type .llcrypt registered [ 936.049986] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 936.793947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 936.810929] alg: No test for adler32 (adler32-zlib) [ 937.843767] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 938.045798] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 939.690954] Key type lgssc registered [ 940.770711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 947.944888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 947.978871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.368981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.435638] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 958.945290] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.954986] LustreError: 34990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 960.849797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.171102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 965.316529] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 971.753039] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 976.938281] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 983.435993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.733954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 989.447113] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 993.976072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 995.863347] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.589192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.595227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.613932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.635605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.639039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.646767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.296450] Lustre: server umount lustre-OST0000 complete [ 1014.762077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1014.766168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1014.779708] Lustre: Skipped 1 previous similar message [ 1014.791714] Lustre: Skipped 2 previous similar messages [ 1019.873980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.880462] Lustre: Skipped 1 previous similar message [ 1020.648207] Lustre: server umount lustre-MDT0000 complete [ 1023.804305] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819119 with bad export cookie 2133387875401767346 [ 1023.805499] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.813577] LustreError: 34970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1024.080723] Lustre: server umount lustre-MDT0001 complete [ 1033.541937] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 1035.808522] Key type lgssc unregistered [ 1036.052623] LNet: 38524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.062429] LNetError: 38524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.074527] LNet: Removed LNI 192.168.206.148@tcp [ 1036.866160] Key type .llcrypt unregistered [ 1036.869847] Key type ._llcrypt unregistered [ 1057.519451] Key type ._llcrypt registered [ 1057.524207] Key type .llcrypt registered [ 1057.684610] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 1059.037613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.064398] alg: No test for adler32 (adler32-zlib) [ 1060.178717] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1060.487149] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 1062.208221] Key type lgssc registered [ 1063.413299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.022069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.036597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.393953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.646246] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1082.854102] LustreError: 39935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.876555] LustreError: 39935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1085.371371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.820295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.890367] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1096.004120] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1100.966641] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1107.341270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1107.678159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.951891] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1116.939347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1121.431330] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1129.014121] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.48@tcp (stopping) [ 1130.318456] Lustre: server umount lustre-MDT0000 complete [ 1132.004394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1132.019615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.037200] LustreError: 39934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.632939] LustreError: 39915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819229 with bad export cookie 7962790402166005317 [ 1133.635421] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.648476] LustreError: 39915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1134.110140] LustreError: 42633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.127892] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.48@tcp (stopping) [ 1134.133684] LustreError: 42633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1137.633050] LustreError: 42633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.638075] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.675565] Lustre: Skipped 2 previous similar messages [ 1137.684851] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1140.091623] Lustre: server umount lustre-MDT0001 complete [ 1160.894514] Lustre: server umount lustre-OST0000 complete [ 1176.319833] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 1179.112708] Key type lgssc unregistered [ 1179.423491] LNet: 43794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.435695] LNetError: 43794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.451716] LNet: Removed LNI 192.168.206.148@tcp [ 1180.221582] Key type .llcrypt unregistered [ 1180.223603] Key type ._llcrypt unregistered [ 1202.312580] Key type ._llcrypt registered [ 1202.318042] Key type .llcrypt registered [ 1202.468992] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 1203.598691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.804047] alg: No test for adler32 (adler32-zlib) [ 1204.905933] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1205.208129] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 1206.920152] Key type lgssc registered [ 1208.078328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1214.821580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1214.863920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.374832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.160560] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1226.728160] LustreError: 45211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.741830] LustreError: 45211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1229.154786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.621041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.777673] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1240.541453] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.303739] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1252.213221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.510346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.144958] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1261.748440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1264.515170] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1270.241921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1270.257170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.282798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1271.374893] Lustre: lustre-OST0000: Not available for connect from 192.168.206.48@tcp (stopping) [ 1272.289304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1272.297897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.648480] Lustre: server umount lustre-OST0000 complete [ 1296.354356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1296.368814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.396211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.414843] Lustre: Skipped 2 previous similar messages [ 1299.234626] Lustre: server umount lustre-MDT0000 complete [ 1302.498833] LustreError: 45207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.521613] LustreError: 45207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1302.839451] LustreError: 45192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819398 with bad export cookie 8130381623218943700 [ 1302.843914] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.851515] LustreError: 45192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1317.344128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1317.595302] Lustre: server umount lustre-MDT0001 complete [ 1327.149969] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 1329.767711] Key type lgssc unregistered [ 1330.100172] LNet: 48924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.106669] LNetError: 48924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.178923] LNet: Removed LNI 192.168.206.148@tcp [ 1331.878678] Key type .llcrypt unregistered [ 1331.886358] Key type ._llcrypt unregistered [ 1352.960669] Key type ._llcrypt registered [ 1352.965664] Key type .llcrypt registered [ 1353.082758] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 1353.788039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1353.818176] alg: No test for adler32 (adler32-zlib) [ 1354.932234] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1355.158989] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 1356.864176] Key type lgssc registered [ 1357.957782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1364.104464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1364.115270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.617711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.908345] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1375.207102] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.218135] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1375.916172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.168991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.571925] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1386.990401] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1391.671641] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.136216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.566450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.981833] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1410.142562] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1410.862742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1418.971581] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1421.350760] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1433.352679] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 4h32m07s [ 1434.693172] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1436.643472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.645214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1436.653368] Lustre: Skipped 2 previous similar messages [ 1436.655534] Lustre: Skipped 1 previous similar message [ 1437.738845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.757328] Lustre: Skipped 3 previous similar messages [ 1441.345782] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.48@tcp (stopping) [ 1442.655611] Lustre: server umount lustre-MDT0000 complete [ 1443.825272] LustreError: 52178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.856756] LustreError: 52178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1446.433917] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1447.452386] LustreError: 50335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.964451] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.989033] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1452.392895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.497963] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1452.753065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.823883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1453.822454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1457.308950] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1460.320674] Lustre: DEBUG MARKER: Wait for df (36751) ... [ 1461.996402] Lustre: DEBUG MARKER: done [ 1468.388290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1468.393729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.396565] LustreError: Skipped 1 previous similar message [ 1468.398045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1468.408791] Lustre: Skipped 1 previous similar message [ 1468.411505] Lustre: Skipped 1 previous similar message [ 1472.455279] Lustre: server umount lustre-OST0000 complete [ 1479.649420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1479.651294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.657368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.670018] Lustre: Skipped 1 previous similar message [ 1479.675749] Lustre: Skipped 3 previous similar messages [ 1482.059772] Lustre: server umount lustre-MDT0000 complete [ 1485.470073] LustreError: 50321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819581 with bad export cookie 5513955104422891813 [ 1485.472320] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.483206] LustreError: 50321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1485.812611] Lustre: server umount lustre-MDT0001 complete [ 1496.666637] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 1499.241267] Key type lgssc unregistered [ 1499.531183] LNet: 55262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.538962] LNetError: 55262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.586503] LNet: Removed LNI 192.168.206.148@tcp [ 1501.441117] Key type .llcrypt unregistered [ 1501.446395] Key type ._llcrypt unregistered [ 1532.395899] Key type ._llcrypt registered [ 1532.404872] Key type .llcrypt registered [ 1532.500704] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 1533.274167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1533.304163] alg: No test for adler32 (adler32-zlib) [ 1534.265326] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1534.453798] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 1536.089365] Key type lgssc registered [ 1537.017292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.972876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1543.008894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.286169] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1544.309712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1544.459226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.311255] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1555.196836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.276085] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1555.300934] Lustre: 57452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1555.534592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.838949] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1564.801741] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1569.122851] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1574.206492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.344499] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1574.537564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.071669] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1580.124072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1584.546827] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1589.102930] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 4h34m44s [ 1590.162939] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1595.157971] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1597.000669] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1597.036409] LustreError: 57458:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1617.488076] Lustre: lustre-MDT0000: Client 6d1412e8-e4a7-4350-a50b-eaab48df9138 (at 192.168.206.48@tcp) reconnecting [ 1637.112465] LustreError: 57458:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1638.955542] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1640.851618] Lustre: DEBUG MARKER: done [ 1646.568891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.573371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1646.575515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1646.577974] Lustre: Skipped 1 previous similar message [ 1646.597588] LustreError: Skipped 1 previous similar message [ 1651.782243] Lustre: server umount lustre-OST0000 complete [ 1656.801471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.814705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.826495] Lustre: Skipped 2 previous similar messages [ 1657.892058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1657.908708] Lustre: Skipped 1 previous similar message [ 1660.940431] Lustre: server umount lustre-MDT0000 complete [ 1662.954320] LustreError: 58490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1662.995120] LustreError: 58490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1664.562406] LustreError: 58194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819760 with bad export cookie 15364838987536542656 [ 1664.565415] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1664.577697] LustreError: 58194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1664.957371] Lustre: server umount lustre-MDT0001 complete [ 1675.406719] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 1678.149421] Key type lgssc unregistered [ 1678.451605] LNet: 61782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.461400] LNetError: 61782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.481651] LNet: Removed LNI 192.168.206.148@tcp [ 1679.389694] Key type .llcrypt unregistered [ 1679.398941] Key type ._llcrypt unregistered [ 1718.920282] Key type ._llcrypt registered [ 1718.922927] Key type .llcrypt registered [ 1719.034265] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 1720.167742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1720.310083] alg: No test for adler32 (adler32-zlib) [ 1721.474781] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1721.784645] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 1723.472167] Key type lgssc registered [ 1724.771473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1727.889722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1727.907445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1729.211841] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1729.254420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1729.403557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.254560] Lustre: Failing over lustre-MDT0000 [ 1730.529749] Lustre: server umount lustre-MDT0000 complete [ 1736.078202] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:37:11 (1787819831) [ 1741.534858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.908878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.017885] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1753.154234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.226084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1753.246327] Lustre: 64583:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1753.453617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.421817] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1764.556939] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1770.015265] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1776.983970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1777.188257] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1777.383178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.705687] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1785.478682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1788.994360] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1794.807983] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1799.651120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1799.658076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.670920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.190105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.194121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.208097] Lustre: Skipped 1 previous similar message [ 1801.216579] Lustre: Skipped 1 previous similar message [ 1801.537699] Lustre: server umount lustre-MDT0000 complete [ 1805.248264] LustreError: 64573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819901 with bad export cookie 8058335688969251538 [ 1805.259890] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.463783] Lustre: server umount lustre-MDT0001 complete [ 1808.182875] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1811.936505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1820.380709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.688629] Lustre: 63507:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787819902/real 1787819902] req@000000006f13c701 x1874664952467840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787819918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1822.742613] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.720164] Lustre: 63506:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787819906/real 1787819906] req@000000007809cdc2 x1874664952467968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787819922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1827.808149] Lustre: 63504:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787819907/real 1787819907] req@000000006aaf4f08 x1874664952468096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787819923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1829.926963] LustreError: 63503:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cbcd9a2d x1874664952468992/t0(0) o250->MGC192.168.206.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1830.321094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.399398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1831.329774] Lustre: 63504:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787819911/real 1787819911] req@00000000b597c7bf x1874664952468352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787819927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1834.510192] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1836.512693] Lustre: 63507:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787819916/real 1787819916] req@0000000040f31eb5 x1874664952468480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787819932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1837.545978] LustreError: 68349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.574820] LustreError: 68349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1841.638127] LustreError: 68350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.664491] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1841.666856] LustreError: 68350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1842.847196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.157160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.150868] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1848.310384] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1853.458420] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1858.026669] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1863.652043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1863.653643] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.655192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.655198] Lustre: Skipped 1 previous similar message [ 1863.676672] Lustre: Skipped 2 previous similar messages [ 1865.766664] Lustre: server umount lustre-MDT0000 complete [ 1868.661990] LustreError: 67490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787819964 with bad export cookie 8058335688969253813 [ 1868.666014] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.678377] LustreError: 67490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1868.771037] LustreError: 68350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.772398] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.801498] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1868.805236] Lustre: Skipped 2 previous similar messages [ 1868.974934] Lustre: server umount lustre-MDT0001 complete [ 1874.548141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1882.215062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.662715] LustreError: 66429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.716206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.782822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1894.762822] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1897.659947] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1900.007700] LustreError: 66429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.020425] LustreError: 66429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1904.063809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.253974] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 1909.556710] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1915.265358] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1922.096810] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1930.144936] Lustre: server umount lustre-MDT0000 complete [ 1930.212356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.215538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1930.229488] LustreError: 66428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.688201] LustreError: 64574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820029 with bad export cookie 8058335688969255192 [ 1933.707577] LustreError: 64574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1933.708306] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.251715] Lustre: server umount lustre-MDT0001 complete [ 1939.851697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1942.712157] Lustre: DEBUG MARKER: files compared the same [ 1954.578703] Lustre: server umount lustre-OST0000 complete [ 1969.888876] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 1972.282504] Key type lgssc unregistered [ 1972.547796] LNet: 74310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.562297] LNetError: 74310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.585779] LNet: Removed LNI 192.168.206.148@tcp [ 1973.509466] Key type .llcrypt unregistered [ 1973.511758] Key type ._llcrypt unregistered [ 1994.461632] Key type ._llcrypt registered [ 1994.464828] Key type .llcrypt registered [ 1994.610955] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 1995.780856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1995.814646] alg: No test for adler32 (adler32-zlib) [ 1996.873520] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 1997.204869] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 1998.872154] Key type lgssc registered [ 2000.293565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.084745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.096837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.497391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2012.589648] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug +malloc all [ 2018.789408] LustreError: 75725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.810186] LustreError: 75725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2019.820864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.084468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.089657] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug +malloc all [ 2030.828954] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2035.258442] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2041.233983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2041.472247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.980775] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug +malloc all [ 2053.425946] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2053.865790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2064.352719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2064.354032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.359637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2064.371872] Lustre: Skipped 1 previous similar message [ 2069.476656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2069.482888] Lustre: Skipped 2 previous similar messages [ 2070.139534] Lustre: server umount lustre-OST0000 complete [ 2074.594807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.595355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2074.618239] Lustre: Skipped 1 previous similar message [ 2074.631305] Lustre: Skipped 1 previous similar message [ 2079.459356] Lustre: server umount lustre-MDT0000 complete [ 2079.714160] LustreError: 78215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.738356] LustreError: 78215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2084.253642] LustreError: 76459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820180 with bad export cookie 3112724477834795213 [ 2084.284208] LustreError: 76459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2084.288210] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.770137] Lustre: server umount lustre-MDT0001 complete [ 2096.856663] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 2099.868326] Key type lgssc unregistered [ 2100.129668] LNet: 79260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.135060] LNetError: 79260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.150967] LNet: Removed LNI 192.168.206.148@tcp [ 2100.871163] Key type .llcrypt unregistered [ 2100.874348] Key type ._llcrypt unregistered [ 2125.346157] Key type ._llcrypt registered [ 2125.347552] Key type .llcrypt registered [ 2125.495852] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 2126.500707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.651058] alg: No test for adler32 (adler32-zlib) [ 2127.800677] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2128.088754] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 2129.779965] Key type lgssc registered [ 2131.141903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2138.991672] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2139.001575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2154.528387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2159.648301] LustreError: 80652:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.148@tcp: failed processing log, type 4: rc = -110 [ 2186.272199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.148033] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2199.541674] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2207.029727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.171161] Lustre: *** cfs_fail_loc=706, val=0*** [ 2207.472897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.553066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2211.396898] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2212.838395] LustreError: 80677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.871557] LustreError: 80677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2217.958485] LustreError: 80678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.606265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.951141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2222.928337] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2230.342129] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.205546] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2238.438434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2238.453529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.467871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.457755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2239.461875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.472776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.510052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.414223] Lustre: server umount lustre-OST0000 complete [ 2249.698849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2249.708948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.722307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.733426] Lustre: Skipped 1 previous similar message [ 2251.746896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.265155] Lustre: server umount lustre-MDT0000 complete [ 2256.865830] LustreError: 81742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.889481] LustreError: 81742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2257.470944] LustreError: 80660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820353 with bad export cookie 3554088619649098037 [ 2257.479483] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.486889] LustreError: 80660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2257.741381] Lustre: server umount lustre-MDT0001 complete [ 2266.908982] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 2269.032546] Key type lgssc unregistered [ 2269.310045] LNet: 84269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.315952] LNetError: 84269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.376397] LNet: Removed LNI 192.168.206.148@tcp [ 2271.149734] Key type .llcrypt unregistered [ 2271.152860] Key type ._llcrypt unregistered [ 2294.766627] Key type ._llcrypt registered [ 2294.771327] Key type .llcrypt registered [ 2294.910795] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 2295.919216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2296.010148] alg: No test for adler32 (adler32-zlib) [ 2297.154176] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2297.383204] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 2299.096166] Key type lgssc registered [ 2300.336302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2308.032228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.070048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.467666] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2321.136618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.650141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2325.846846] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2327.011129] LustreError: 86374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.030822] LustreError: 86374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2331.124363] LustreError: 86377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.154382] LustreError: 86377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2334.034488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2334.310200] LustreError: 86374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.329137] LustreError: 86374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2334.355950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.702680] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2339.817813] LustreError: 86378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.835943] LustreError: 86378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2344.932385] LustreError: 86377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.951880] LustreError: 86377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2347.753452] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2347.765462] Lustre: MGS: Client 03c0494d-89f2-4864-9803-7cb812ff2e00 (at 0@lo) reconnecting [ 2347.778124] LustreError: 86388:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2347.979243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.062633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2351.829304] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2368.481148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.485959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.490662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.519486] Lustre: Skipped 1 previous similar message [ 2371.933911] Lustre: server umount lustre-OST0000 complete [ 2378.732800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.734524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.744502] Lustre: Skipped 1 previous similar message [ 2378.746731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.775430] Lustre: Skipped 2 previous similar messages [ 2381.931209] Lustre: server umount lustre-MDT0000 complete [ 2383.843311] LustreError: 86378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.864448] LustreError: 86378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2386.197479] Lustre: server umount lustre-MDT0001 complete [ 2389.983761] Lustre: server umount MGS complete [ 2400.658741] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:48:15 (1787820495) [ 2428.226053] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 2438.293059] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 2473.258873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.203747] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2488.527201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.721858] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2488.767061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2488.842209] Lustre: lustre-MDT0001: new disk, initializing [ 2488.914866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2488.944682] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2488.957708] Lustre: 93554:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2488.964816] LustreError: 93554:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2494.108149] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2504.481109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2504.801598] Lustre: lustre-OST0000: new disk, initializing [ 2504.809503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2504.828554] Lustre: 94321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2504.944251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.557255] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2513.442652] Lustre: 94755:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f43a53fb x1874665556451072/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787820625 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2520.788938] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2527.197741] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.218344] Lustre: MGS: Client 9d829fd5-0406-4ffd-8659-e96ebe1b7c36 (at 0@lo) reconnecting [ 2527.249065] LustreError: 93540:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2527.297922] Lustre: 92823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2527.377096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2527.462503] Lustre: lustre-MDT0000: new disk, initializing [ 2527.573274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.603484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2527.688399] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2528.677390] Lustre: 95389:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b8aa498d x1874665556457728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787820640 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2532.725656] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2533.862210] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2537.769302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2537.782477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2537.931026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2563.553257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.566551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.583929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.627249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2566.637732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.662752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.417874] Lustre: server umount lustre-OST0000 complete [ 2572.259033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.264479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.271683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.278115] Lustre: Skipped 1 previous similar message [ 2572.301528] Lustre: Skipped 3 previous similar messages [ 2578.091767] Lustre: server umount lustre-MDT0000 complete [ 2588.139861] Lustre: server umount lustre-MDT0001 complete [ 2592.484447] Lustre: server umount MGS complete [ 2603.864943] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:51:38 (1787820698) [ 2621.619465] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 2624.195280] Key type lgssc unregistered [ 2624.500631] LNet: 97887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2624.508992] LNetError: 97887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2624.532586] LNet: Removed LNI 192.168.206.148@tcp [ 2625.477608] Key type .llcrypt unregistered [ 2625.480853] Key type ._llcrypt unregistered [ 2641.062808] Key type ._llcrypt registered [ 2641.065806] Key type .llcrypt registered [ 2641.192652] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 2642.084176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2642.167291] alg: No test for adler32 (adler32-zlib) [ 2643.244850] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2643.532957] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 2645.256263] Key type lgssc registered [ 2646.355709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2653.420848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2653.449507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.928446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.579821] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2665.446676] LustreError: 99110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.455254] LustreError: 99110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2667.838065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.208845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2672.525397] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2678.868935] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2683.831263] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2690.075682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.388248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.031754] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2700.661584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2703.300658] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2711.010895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.019642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.031947] Lustre: Skipped 1 previous similar message [ 2711.034303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2711.037752] Lustre: Skipped 1 previous similar message [ 2716.137352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.142538] Lustre: Skipped 1 previous similar message [ 2717.077500] Lustre: server umount lustre-OST0000 complete [ 2721.250350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.260214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.263501] Lustre: Skipped 1 previous similar message [ 2726.373897] LustreError: 99111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.388150] LustreError: 99111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2726.527876] Lustre: server umount lustre-MDT0000 complete [ 2730.012019] LustreError: 99091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820826 with bad export cookie 10424845363701532865 [ 2730.021266] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.036335] LustreError: 99091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2730.377466] Lustre: server umount lustre-MDT0001 complete [ 2739.602544] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 2742.095554] Key type lgssc unregistered [ 2742.390379] LNet: 102644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2742.397505] LNetError: 102644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2742.412382] LNet: Removed LNI 192.168.206.148@tcp [ 2743.091718] Key type .llcrypt unregistered [ 2743.094068] Key type ._llcrypt unregistered [ 2757.091919] Key type ._llcrypt registered [ 2757.094205] Key type .llcrypt registered [ 2757.209290] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 2758.238931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.381037] alg: No test for adler32 (adler32-zlib) [ 2759.552705] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2759.821067] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 2761.568164] Key type lgssc registered [ 2762.795660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2768.823254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2768.833530] LustreError: 103718:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2768.867475] LustreError: 103724:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2768.874717] LustreError: 103718:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4972 [ 2768.878360] LustreError: 103718:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2768.886449] LustreError: 103718:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2768.892426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.368421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.556219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.819070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.168924] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2785.198146] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing lsmod [ 2787.386431] LustreError: 104919:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2787.447980] LustreError: 104931:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2787.457110] LustreError: 104919:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4936 [ 2787.462799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2787.467457] LustreError: 104919:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2787.467506] LustreError: 104919:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2787.833703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.930416] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.934799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2792.944067] Lustre: Skipped 2 previous similar messages [ 2792.960751] Lustre: Skipped 2 previous similar messages [ 2796.457465] Lustre: server umount lustre-MDT0000 complete [ 2798.050807] LustreError: 103759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.068325] LustreError: 103759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2799.564877] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820895 with bad export cookie 8827477462083414749 [ 2799.569560] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2799.579600] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2799.588077] LustreError: 103757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.589158] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.590123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2799.844355] Lustre: server umount lustre-MDT0001 complete [ 2803.077741] Lustre: server umount lustre-OST0000 complete [ 2810.378746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.874663] LustreError: 105997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.947780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.853883] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2815.976329] LustreError: 105998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.093598] LustreError: 105997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.116200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.451451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2825.986944] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2832.450401] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2836.826360] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2841.858744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2842.117120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.126789] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2854.345540] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2856.502961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2866.658446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.664301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2866.669895] Lustre: Skipped 1 previous similar message [ 2866.690912] Lustre: Skipped 1 previous similar message [ 2871.789654] Lustre: server umount lustre-OST0000 complete [ 2876.901205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.928817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2881.936544] Lustre: server umount lustre-MDT0000 complete [ 2882.020958] LustreError: 108189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.043466] LustreError: 108189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2885.576764] LustreError: 105977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787820981 with bad export cookie 8827477462083415428 [ 2885.600973] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2885.602771] LustreError: 105977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2886.042515] Lustre: server umount lustre-MDT0001 complete [ 2896.383538] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 2899.603749] Key type lgssc unregistered [ 2899.961971] LNet: 109531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2899.972329] LNetError: 109531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.984596] LNet: Removed LNI 192.168.206.148@tcp [ 2900.870223] Key type .llcrypt unregistered [ 2900.872181] Key type ._llcrypt unregistered [ 2922.781484] Key type ._llcrypt registered [ 2922.784438] Key type .llcrypt registered [ 2922.885604] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 2923.922173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2923.981056] alg: No test for adler32 (adler32-zlib) [ 2925.053786] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 2925.317938] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 2927.000158] Key type lgssc registered [ 2928.199548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2935.554487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2935.575627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.004282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.600811] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2947.562602] LustreError: 110940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.588268] LustreError: 110940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2948.541381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.999205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.051788] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2959.987358] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2964.870445] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2971.597690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.888431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.939581] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 2982.147100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2985.623769] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2990.239563] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2993.476727] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2997.737780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2997.769232] LustreError: 113270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.876321] Lustre: server umount lustre-OST0000 complete [ 3002.851744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.859340] Lustre: Skipped 1 previous similar message [ 3002.864638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.868812] Lustre: Skipped 1 previous similar message [ 3007.695913] Lustre: server umount lustre-MDT0000 complete [ 3007.971484] LustreError: 110935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.985119] LustreError: 110935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3011.186924] LustreError: 110922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821107 with bad export cookie 753246228625813480 [ 3011.187951] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.203584] LustreError: 110922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3011.613953] Lustre: server umount lustre-MDT0001 complete [ 3022.134453] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 3024.657708] Key type lgssc unregistered [ 3024.983657] LNet: 114582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3024.989728] LNetError: 114582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3025.016535] LNet: Removed LNI 192.168.206.148@tcp [ 3025.894341] Key type .llcrypt unregistered [ 3025.898244] Key type ._llcrypt unregistered [ 3042.814838] Key type ._llcrypt registered [ 3042.817056] Key type .llcrypt registered [ 3042.921923] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 3044.110230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3044.136571] alg: No test for adler32 (adler32-zlib) [ 3045.175386] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3045.393552] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 3047.032944] Key type lgssc registered [ 3047.954914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3054.485578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3054.505252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.801766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.959682] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3065.319540] LustreError: 115802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.344837] LustreError: 115802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3066.615821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.877956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.529352] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3078.297772] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3083.917641] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3090.153167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.567422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.172508] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3100.878086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3103.416540] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3112.417779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3112.438582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.465575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3113.957298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3113.964721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.984211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3116.514581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3116.529355] Lustre: Skipped 1 previous similar message [ 3118.684823] Lustre: server umount lustre-OST0000 complete [ 3123.179497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3123.189550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.214491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.756319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.852917] Lustre: server umount lustre-MDT0000 complete [ 3131.883478] LustreError: 115801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.918583] LustreError: 115801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3134.051892] LustreError: 115783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821230 with bad export cookie 1494489376648250830 [ 3134.076293] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.080206] LustreError: 115783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3134.592730] Lustre: server umount lustre-MDT0001 complete [ 3145.664069] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 3148.768990] Key type lgssc unregistered [ 3148.943375] LNet: 119335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.947328] LNetError: 119335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3148.960643] LNet: Removed LNI 192.168.206.148@tcp [ 3149.740834] Key type .llcrypt unregistered [ 3149.747106] Key type ._llcrypt unregistered [ 3175.510340] Key type ._llcrypt registered [ 3175.514777] Key type .llcrypt registered [ 3175.590228] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 3176.313445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.332306] alg: No test for adler32 (adler32-zlib) [ 3177.377362] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3177.637483] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 3179.336284] Key type lgssc registered [ 3180.119859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.064135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3186.083750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3187.404508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.107317] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3196.907309] LustreError: 120767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.919189] LustreError: 120767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3198.647408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.952186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.231941] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3208.957774] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3215.532235] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3223.122289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3223.337984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.434809] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3232.559962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3236.265212] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3242.863263] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3245.809855] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3245.813823] Lustre: Skipped 2 previous similar messages [ 3249.244280] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3249.248876] Lustre: Skipped 2 previous similar messages [ 3252.333439] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3252.353577] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3257.874982] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.48@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3257.882670] Lustre: Skipped 2 previous similar messages [ 3261.628703] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.48@tcp 192.168.0.[2,10]@tcp [ 3265.395727] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.48/24@tcp in log lustre-MDT0000 [ 3265.407607] Lustre: Skipped 2 previous similar messages [ 3268.292253] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.48/24@tcp [ 3268.297127] Lustre: Skipped 1 previous similar message [ 3273.698288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.699400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.702635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.717908] Lustre: Skipped 1 previous similar message [ 3278.825818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.838039] Lustre: Skipped 2 previous similar messages [ 3279.698437] Lustre: server umount lustre-OST0000 complete [ 3283.939184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.948237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.950367] Lustre: Skipped 1 previous similar message [ 3283.965466] Lustre: Skipped 1 previous similar message [ 3289.060758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.078671] Lustre: Skipped 1 previous similar message [ 3289.335519] Lustre: server umount lustre-MDT0000 complete [ 3292.875914] LustreError: 124395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821389 with bad export cookie 9806309982941320996 [ 3292.883343] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.890436] LustreError: 124395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.219276] Lustre: server umount lustre-MDT0001 complete [ 3303.750133] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 3306.625701] Key type lgssc unregistered [ 3306.825713] LNet: 125605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.835317] LNetError: 125605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3307.888396] LNet: Removed LNI 192.168.206.148@tcp [ 3308.701199] Key type .llcrypt unregistered [ 3308.703665] Key type ._llcrypt unregistered [ 3335.752101] Key type ._llcrypt registered [ 3335.754281] Key type .llcrypt registered [ 3335.851532] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 3336.978661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3337.000647] alg: No test for adler32 (adler32-zlib) [ 3338.031366] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3338.252801] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 3339.896364] Key type lgssc registered [ 3341.125194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3348.456459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3348.477530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3349.937456] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3349.958288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.974938] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.48/24@tcp [ 3354.694720] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3360.227767] LustreError: 127166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.264702] LustreError: 127166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3363.671610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.175050] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3364.199840] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.48/24@tcp [ 3364.283578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.838770] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3375.868222] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3380.992147] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3386.584087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.947577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.607255] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3395.137288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3400.397405] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3410.404966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.411191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3410.426892] Lustre: Skipped 1 previous similar message [ 3410.427297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3410.447233] Lustre: Skipped 1 previous similar message [ 3415.532927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3415.542493] Lustre: Skipped 1 previous similar message [ 3416.430259] Lustre: server umount lustre-OST0000 complete [ 3420.642424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3420.643776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.651571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3420.670711] Lustre: Skipped 1 previous similar message [ 3425.761066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.771619] Lustre: Skipped 2 previous similar messages [ 3426.378103] Lustre: server umount lustre-MDT0000 complete [ 3429.860494] LustreError: 127147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821526 with bad export cookie 6116917608251866987 [ 3429.864130] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.880829] LustreError: 127147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3430.341418] Lustre: server umount lustre-MDT0001 complete [ 3438.569210] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 3440.733885] Key type lgssc unregistered [ 3441.012663] LNet: 130755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3441.024707] LNetError: 130755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3441.053971] LNet: Removed LNI 192.168.206.148@tcp [ 3441.897653] Key type .llcrypt unregistered [ 3441.901115] Key type ._llcrypt unregistered [ 3466.294514] Key type ._llcrypt registered [ 3466.297707] Key type .llcrypt registered [ 3466.440282] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 3481.803421] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 3482.866038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3482.975414] alg: No test for adler32 (adler32-zlib) [ 3484.018857] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3484.238621] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 3485.945405] Key type lgssc registered [ 3486.845431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3517.152177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3517.171753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.395195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3518.415503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3518.473817] Lustre: lustre-MDT0000: new disk, initializing [ 3518.517671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.539177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3522.462323] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3530.745912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.824222] Lustre: 134154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3530.876509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3530.886211] Lustre: Skipped 1 previous similar message [ 3530.928768] Lustre: lustre-MDT0001: new disk, initializing [ 3530.981857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.011628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3531.017682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3535.058767] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3543.883882] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3548.975436] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3554.930203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.274428] Lustre: lustre-OST0000: new disk, initializing [ 3555.279790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3555.290632] Lustre: 136123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3555.421665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.011282] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3562.549333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3562.558333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3562.662131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3569.278589] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3573.271976] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3573.431984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3577.242449] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3577.415039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3583.457077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3583.463822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.476038] Lustre: Skipped 1 previous similar message [ 3583.477625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3583.496942] Lustre: Skipped 1 previous similar message [ 3588.585467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.592062] Lustre: Skipped 1 previous similar message [ 3593.184130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3593.353332] Lustre: server umount lustre-OST0000 complete [ 3597.796759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3597.801714] LustreError: Skipped 1 previous similar message [ 3597.808333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.823890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.822090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.512790] Lustre: server umount lustre-MDT0000 complete [ 3605.508930] LustreError: 134147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821701 with bad export cookie 990104351818967761 [ 3605.509388] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3605.518603] LustreError: 134147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3605.696794] Lustre: server umount lustre-MDT0001 complete [ 3612.960111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.361577] LustreError: 138339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.383437] LustreError: 138339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3613.457345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.026776] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3618.789341] LustreError: 138340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3622.885851] LustreError: 138339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.306476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.566828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3628.558578] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3634.662788] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3639.094668] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3644.421359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.658458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.968887] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3657.005149] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3661.551206] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3661.748226] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3665.977355] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3666.178764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3674.485708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.659484] Lustre: lustre-OST0001: new disk, initializing [ 3674.663714] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3674.672944] Lustre: 141711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3674.769530] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.964440] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3680.900961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3680.921195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3681.045263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3686.624705] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3686.823607] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3690.474629] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3690.655512] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3695.639594] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e1904b89800.ost_server_uuid 50 [ 3696.818460] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e1904b89800.ost_server_uuid in FULL state after 0 sec [ 3706.851410] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3706.851894] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.862569] LustreError: Skipped 1 previous similar message [ 3706.865504] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3706.873301] Lustre: Skipped 1 previous similar message [ 3706.900383] Lustre: Skipped 3 previous similar messages [ 3708.922846] Lustre: server umount lustre-OST0001 complete [ 3711.977670] LustreError: 140161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3712.005141] LustreError: 140161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3714.529743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3714.536533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.550735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.106266] LustreError: 140162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3717.128346] LustreError: 140162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3727.330771] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3727.339041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3727.339765] LustreError: 140162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.339773] LustreError: 140162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3727.404952] Lustre: Skipped 6 previous similar messages [ 3727.592386] Lustre: server umount lustre-OST0000 complete [ 3731.939820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3731.951131] LustreError: Skipped 1 previous similar message [ 3731.954333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.960918] Lustre: Skipped 1 previous similar message [ 3737.436471] Lustre: server umount lustre-MDT0000 complete [ 3740.361080] LustreError: 141710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787821836 with bad export cookie 990104351818968650 [ 3740.367118] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3740.368647] LustreError: 141710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3740.777093] Lustre: server umount lustre-MDT0001 complete [ 3758.012300] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 3760.796949] Key type lgssc unregistered [ 3761.107028] LNet: 144715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3761.112632] LNetError: 144715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3761.131681] LNet: Removed LNI 192.168.206.148@tcp [ 3762.068856] Key type .llcrypt unregistered [ 3762.072133] Key type ._llcrypt unregistered [ 3796.020240] Key type ._llcrypt registered [ 3796.022383] Key type .llcrypt registered [ 3796.124965] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 3810.680739] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 3812.065914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3812.109466] alg: No test for adler32 (adler32-zlib) [ 3813.179754] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 3813.405347] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 3815.090549] Key type lgssc registered [ 3816.107449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3849.162645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3849.190970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.469742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3850.508127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3850.640283] Lustre: lustre-MDT0000: new disk, initializing [ 3850.773505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.799948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3855.773479] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3866.008722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.087269] Lustre: 148869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.134607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.138332] Lustre: Skipped 1 previous similar message [ 3866.218483] Lustre: lustre-MDT0001: new disk, initializing [ 3866.284981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.321348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3866.341534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3870.433413] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3878.300807] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3883.435240] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3890.365397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3890.758465] Lustre: lustre-OST0000: new disk, initializing [ 3890.769441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3890.781703] Lustre: 150835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3890.925633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3897.728798] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3899.009637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3899.018690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3899.187447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3907.562438] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3912.793685] Lustre: Failing over lustre-OST0000 [ 3913.000691] Lustre: server umount lustre-OST0000 complete [ 3930.083663] Lustre: 147092:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787822010/real 1787822010] req@000000008b8eecc0 x1874667146086272/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787822026 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3930.085971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.103118] Lustre: 147092:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3930.114083] Lustre: Skipped 1 previous similar message [ 3931.105790] Lustre: 147094:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787822010/real 1787822010] req@00000000ddf7cf23 x1874667146086400/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787822026 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3931.153083] Lustre: 147094:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3931.554145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.931375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.948987] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3935.203971] Lustre: 147092:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787822015/real 1787822015] req@00000000df3de95e x1874667146087680/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787822031 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3937.926035] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3940.075114] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3940.932606] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3940.932870] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3941.615843] Lustre: Failing over lustre-MDT0000 [ 3941.850868] Lustre: server umount lustre-MDT0000 complete [ 3942.624093] Lustre: 147092:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787822021/real 1787822021] req@00000000a89c5a70 x1874667146088704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787822037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3942.670522] Lustre: 147092:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3943.413844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3943.423203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.445772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.466261] Lustre: Skipped 1 previous similar message [ 3946.689060] Lustre: 147093:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787822026/real 1787822026] req@000000006a630135 x1874667146089856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787822042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3950.563847] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3950.576704] LustreError: 148880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3951.127410] LustreError: 148876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.691045] LustreError: 150029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.719861] LustreError: 150029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3960.047636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.154438] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.447834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.496151] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3961.372416] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3965.158774] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 3965.960595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3965.966872] Lustre: Skipped 1 previous similar message [ 3965.994195] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3966.032763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3976.168325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3976.176834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.184818] Lustre: Skipped 1 previous similar message [ 3979.216854] Lustre: server umount lustre-OST0000 complete [ 3986.402076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.405353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3986.422364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.430859] Lustre: Skipped 2 previous similar messages [ 3988.593957] Lustre: server umount lustre-MDT0000 complete [ 3991.532734] LustreError: 150029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.554918] LustreError: 150029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3993.080206] LustreError: 148860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822089 with bad export cookie 8961379648060846337 [ 3993.093898] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3993.094937] LustreError: 148860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3993.663043] Lustre: server umount lustre-MDT0001 complete [ 4004.640698] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 4007.068210] Key type lgssc unregistered [ 4007.405483] LNet: 154492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4007.432247] LNetError: 154492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4007.448759] LNet: Removed LNI 192.168.206.148@tcp [ 4008.427263] Key type .llcrypt unregistered [ 4008.431137] Key type ._llcrypt unregistered [ 4041.803635] Key type ._llcrypt registered [ 4041.805970] Key type .llcrypt registered [ 4041.890984] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 4042.738545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4042.753265] alg: No test for adler32 (adler32-zlib) [ 4043.820879] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 4044.169673] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 4045.824292] Key type lgssc registered [ 4046.889837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4053.795648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4053.820169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.198609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.829719] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4065.767904] LustreError: 156013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.783524] LustreError: 156013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4068.090505] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4070.883486] LustreError: 156014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.818011] LustreError: 156042:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4071.825200] LustreError: 156042:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4071.854642] LustreError: 156042:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4072.222411] Lustre: server umount lustre-MDT0000 complete [ 4086.395168] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 4095.547724] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 4129.976406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.157972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4130.201765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4130.311095] Lustre: lustre-MDT0000: new disk, initializing [ 4130.401331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.417043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4134.893350] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4144.096102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4144.207669] Lustre: 159346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4144.258922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4144.263417] Lustre: Skipped 1 previous similar message [ 4144.323110] Lustre: lustre-MDT0001: new disk, initializing [ 4144.408451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4144.440657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4144.451725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4148.471476] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4157.126900] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4162.183250] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4167.716693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4167.997582] Lustre: lustre-OST0000: new disk, initializing [ 4168.005259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4168.012312] Lustre: 161314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4168.093621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.350188] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4176.940516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4176.951983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4177.028091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4181.010492] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4184.330390] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4185.756518] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:18:01 (1787822281) [ 4405.750169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4405.758909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.793662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.266838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.267815] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4410.939895] Lustre: server umount lustre-MDT0000 complete [ 4412.384809] LustreError: 159357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.406848] LustreError: 159357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4414.454168] LustreError: 159336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822510 with bad export cookie 12268123968928601576 [ 4414.460378] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.462975] LustreError: 159336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4417.505308] LustreError: 161656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.510322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.544533] Lustre: Skipped 1 previous similar message [ 4417.552397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4417.558172] Lustre: Skipped 2 previous similar messages [ 4419.553214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4420.842815] Lustre: server umount lustre-MDT0001 complete [ 4430.610684] Lustre: server umount lustre-OST0000 complete [ 4438.565968] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 4446.710433] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 4480.065543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.245623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4480.268195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4480.395194] Lustre: lustre-MDT0000: new disk, initializing [ 4480.461111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.477166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4484.825331] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4495.568757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.731810] Lustre: 165978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4495.786440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4495.789155] Lustre: Skipped 1 previous similar message [ 4495.844598] Lustre: lustre-MDT0001: new disk, initializing [ 4495.920746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.957939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4495.979427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4501.212520] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4511.848637] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4517.829842] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4524.525106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4524.702644] Lustre: lustre-OST0000: new disk, initializing [ 4524.705712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4524.716272] Lustre: 167944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4524.808461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4530.278444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4530.292038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4530.447353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4531.018800] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4539.603849] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4543.761259] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4544.003720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4547.741866] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4547.929356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4551.140507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4551.143799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.151313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4551.176334] Lustre: Skipped 1 previous similar message [ 4555.646127] Lustre: server umount lustre-OST0000 complete [ 4561.378878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.402619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.407089] Lustre: Skipped 2 previous similar messages [ 4564.880870] Lustre: server umount lustre-MDT0000 complete [ 4566.504576] LustreError: 165988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4566.525385] LustreError: 165988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4568.110423] LustreError: 165968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822664 with bad export cookie 12268123968928760406 [ 4568.117720] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4568.121399] LustreError: 165968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4568.475420] Lustre: server umount lustre-MDT0001 complete [ 4575.454117] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:24:30 (1787822670) [ 4588.564339] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 4596.338935] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 4628.699355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4629.023115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4629.058047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4629.134956] Lustre: lustre-MDT0000: new disk, initializing [ 4629.201622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.218843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4633.516433] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4643.015652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.118945] Lustre: 172698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4643.357961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4647.148448] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4656.097898] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.522689] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4667.604455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.903674] Lustre: lustre-OST0000: new disk, initializing [ 4667.908289] Lustre: Skipped 1 previous similar message [ 4667.914576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4667.926458] Lustre: Skipped 2 previous similar messages [ 4667.938557] Lustre: 174674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4674.554536] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4675.651903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4675.661109] Lustre: Skipped 1 previous similar message [ 4675.676168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4675.859986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4684.295475] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.425944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4691.430578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.431740] LustreError: Skipped 1 previous similar message [ 4691.433409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4691.442623] Lustre: Skipped 2 previous similar messages [ 4696.359101] Lustre: server umount lustre-OST0000 complete [ 4699.617635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4705.716673] Lustre: server umount lustre-MDT0000 complete [ 4706.805902] LustreError: 172709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.836970] LustreError: 172709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4709.231217] LustreError: 172688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822805 with bad export cookie 12268123968928761288 [ 4709.239455] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.245993] LustreError: 172688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4709.632539] Lustre: server umount lustre-MDT0001 complete [ 4717.124479] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:26:52 (1787822812) [ 4732.505816] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 4741.747675] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 4774.418411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.667946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4774.714222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4774.819573] Lustre: lustre-MDT0000: new disk, initializing [ 4774.901323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.908081] Lustre: Skipped 2 previous similar messages [ 4774.931651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4778.788316] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4788.495636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.648301] Lustre: 178993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4788.979660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4793.366244] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4801.812986] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4807.295401] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4814.271242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.555518] Lustre: 180969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4820.648726] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4822.088407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4822.224478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4830.185370] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4837.857683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4837.878582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.891280] Lustre: Skipped 2 previous similar messages [ 4837.896772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4837.905943] Lustre: Skipped 5 previous similar messages [ 4842.530420] Lustre: server umount lustre-OST0000 complete [ 4851.859825] Lustre: server umount lustre-MDT0000 complete [ 4853.216992] LustreError: 179005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.223695] LustreError: 179005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4854.805126] LustreError: 178987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787822950 with bad export cookie 12268123968928762373 [ 4854.805383] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.817479] LustreError: 178987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4855.243758] Lustre: server umount lustre-MDT0001 complete [ 4865.526826] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 4869.283426] Key type lgssc unregistered [ 4869.575941] LNet: 182829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4869.593251] LNetError: 182829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4869.615671] LNet: Removed LNI 192.168.206.148@tcp [ 4870.421979] Key type .llcrypt unregistered [ 4870.425066] Key type ._llcrypt unregistered [ 4892.805275] Key type ._llcrypt registered [ 4892.807880] Key type .llcrypt registered [ 4892.975778] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 4894.281669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4894.296111] alg: No test for adler32 (adler32-zlib) [ 4895.438902] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 4895.784844] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 4897.488172] Key type lgssc registered [ 4898.725054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4906.337747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4906.353734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.853349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.873384] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4917.218290] LustreError: 184224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.250676] LustreError: 184224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4920.582863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4920.898444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4925.404667] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4932.953269] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4939.168776] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4945.699139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4946.080554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.451449] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 4952.243379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4958.274356] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4986.341602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4986.349876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.351601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4986.366108] Lustre: Skipped 1 previous similar message [ 4986.375532] Lustre: Skipped 1 previous similar message [ 4990.187462] Lustre: server umount lustre-OST0000 complete [ 4996.579033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.593344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.597827] Lustre: Skipped 1 previous similar message [ 4999.749678] Lustre: server umount lustre-MDT0000 complete [ 5001.697665] LustreError: 185231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.718093] LustreError: 185231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5002.650855] LustreError: 184205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823098 with bad export cookie 11108104548644730435 [ 5002.654905] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5002.665380] LustreError: 184205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5003.001724] Lustre: server umount lustre-MDT0001 complete [ 5011.785462] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 5014.612903] Key type lgssc unregistered [ 5015.026903] LNet: 187782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5015.034835] LNetError: 187782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5015.057614] LNet: Removed LNI 192.168.206.148@tcp [ 5015.826615] Key type .llcrypt unregistered [ 5015.830169] Key type ._llcrypt unregistered [ 5035.331281] Key type ._llcrypt registered [ 5035.333940] Key type .llcrypt registered [ 5035.425661] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 5036.523433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5036.553450] alg: No test for adler32 (adler32-zlib) [ 5037.623501] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 5037.911932] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 5039.528203] Key type lgssc registered [ 5040.467126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.650409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.671793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.162511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.391756] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5059.560800] LustreError: 189181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.576938] LustreError: 189181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5060.813186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.215856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5066.096184] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5072.805436] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5078.597597] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5084.061687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5084.358885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.940736] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5094.586249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5097.633350] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5104.688691] Lustre: lustre-OST0000: Not available for connect from 192.168.206.48@tcp (stopping) [ 5105.122509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.125336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.008202] Lustre: server umount lustre-OST0000 complete [ 5115.897337] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e1904b8f800.ost_server_uuid 50 [ 5117.435660] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5130.723725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.725201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5130.744594] Lustre: Skipped 2 previous similar messages [ 5130.755775] Lustre: Skipped 2 previous similar messages [ 5135.843223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5135.849835] Lustre: Skipped 2 previous similar messages [ 5136.975950] Lustre: server umount lustre-MDT0000 complete [ 5140.107727] LustreError: 189159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823236 with bad export cookie 13955237054176458800 [ 5140.114033] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.123742] LustreError: 189159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5140.461598] Lustre: server umount lustre-MDT0001 complete [ 5148.324896] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:34:03 (1787823243) [ 5154.868671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.246481] LustreError: 193096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.317818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.322273] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5160.420411] LustreError: 193097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.540446] LustreError: 193096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.016156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.354483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5171.884783] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5179.834853] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5186.173782] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5192.648179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5193.104910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.550099] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5200.315771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5206.764801] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5213.264096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5213.448580] Lustre: lustre-OST0001: new disk, initializing [ 5213.452764] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5213.462043] Lustre: 195938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5213.552080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.252444] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5221.992826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5222.004846] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5222.179907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5229.226938] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.618111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.634148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5232.640814] Lustre: Skipped 1 previous similar message [ 5237.740524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5237.747151] Lustre: Skipped 2 previous similar messages [ 5238.044788] Lustre: server umount lustre-OST0000 complete [ 5239.830857] LustreError: 196385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.913374] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5244.101230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5247.972170] LustreError: 196385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.005908] LustreError: 196385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5248.026686] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5248.207713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5263.330744] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5263.341379] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.351161] Lustre: Skipped 1 previous similar message [ 5263.358646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5263.362869] Lustre: Skipped 1 previous similar message [ 5268.092242] Lustre: server umount lustre-OST0001 complete [ 5273.584255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.607693] Lustre: Skipped 2 previous similar messages [ 5278.089194] Lustre: server umount lustre-MDT0000 complete [ 5278.691895] LustreError: 193092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.722702] LustreError: 193092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5281.918179] LustreError: 193076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823378 with bad export cookie 13955237054176459871 [ 5281.921347] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5281.927208] LustreError: 193076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5282.353345] Lustre: server umount lustre-MDT0001 complete [ 5303.058563] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:36:38 (1787823398) [ 5308.954156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.256432] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5309.292149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5309.444338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.159544] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5320.743550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.826490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5320.856932] Lustre: 199374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5324.822692] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5330.358026] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5334.755032] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5340.013125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.229458] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5345.434620] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5349.037135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5352.071473] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5357.660343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5357.748091] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5363.309272] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5364.313671] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5369.079688] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5374.437249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.438488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5374.494875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5374.495268] LustreError: Skipped 1 previous similar message [ 5374.500299] Lustre: Skipped 2 previous similar messages [ 5377.729629] Lustre: server umount lustre-OST0000 complete [ 5379.553967] LustreError: 201220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.574467] LustreError: 201220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5394.914412] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.918240] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.928657] Lustre: Skipped 2 previous similar messages [ 5400.908953] Lustre: server umount lustre-OST0001 complete [ 5410.274551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5410.281263] Lustre: Skipped 8 previous similar messages [ 5411.288802] Lustre: server umount lustre-MDT0000 complete [ 5414.528612] LustreError: 199364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823510 with bad export cookie 13955237054176461278 [ 5414.534316] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5414.534574] LustreError: 199364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5414.813742] Lustre: server umount lustre-MDT0001 complete [ 5436.286753] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:38:51 (1787823531) [ 5451.953873] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 5461.170227] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 5488.510676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.681626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5488.707386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5488.770236] Lustre: lustre-MDT0000: new disk, initializing [ 5488.817390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.821290] Lustre: Skipped 3 previous similar messages [ 5488.836186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5492.118701] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5500.451297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.543953] Lustre: 207474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5500.578779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5500.583187] Lustre: Skipped 1 previous similar message [ 5500.647174] Lustre: lustre-MDT0001: new disk, initializing [ 5500.768524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5500.788567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5503.979294] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5512.759471] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5517.730745] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5523.212126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5523.467092] Lustre: lustre-OST0000: new disk, initializing [ 5523.473934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5523.488195] Lustre: 209438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5528.699088] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5532.210540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5532.217471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5532.292503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5536.398299] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5540.434178] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5540.672541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5545.181553] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5545.424563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5548.005041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5548.006991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.020491] LustreError: Skipped 1 previous similar message [ 5548.021755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5548.035450] Lustre: Skipped 3 previous similar messages [ 5548.045747] Lustre: Skipped 1 previous similar message [ 5561.312131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5561.680806] Lustre: server umount lustre-OST0000 complete [ 5567.457380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5571.914621] Lustre: server umount lustre-MDT0000 complete [ 5573.606160] LustreError: 209811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.633755] LustreError: 209811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5575.511597] LustreError: 207464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823671 with bad export cookie 13955237054176463049 [ 5575.515509] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.524155] LustreError: 207464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5575.981502] Lustre: server umount lustre-MDT0001 complete [ 5583.626240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.007790] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5595.196640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.545694] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5605.587517] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5610.565215] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5616.557423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5616.855689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.865429] Lustre: Skipped 4 previous similar messages [ 5622.726788] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5628.631168] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5632.859176] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5633.138303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5637.035563] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5637.273931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5639.655540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5639.655870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.689580] Lustre: Skipped 2 previous similar messages [ 5645.293598] Lustre: server umount lustre-OST0000 complete [ 5650.963760] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5655.412870] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5683.885384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5689.754990] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5695.536496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.555610] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.557899] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5696.731943] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5703.511840] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5710.817987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5710.841176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5710.849615] Lustre: Skipped 12 previous similar messages [ 5713.581742] Lustre: server umount lustre-OST0000 complete [ 5731.296143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5731.499378] Lustre: server umount lustre-MDT0000 complete [ 5735.387959] LustreError: 215763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823831 with bad export cookie 13955237054176463931 [ 5735.389245] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5735.397415] LustreError: 215763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5735.799041] Lustre: server umount lustre-MDT0001 complete [ 5743.844829] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:43:58 (1787823838) [ 5750.252965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.585404] LustreError: 217625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.609713] LustreError: 217625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5754.513074] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5761.166821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.522058] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5772.513882] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5777.094840] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5782.538279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5788.010106] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5794.346297] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5798.327636] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5798.518783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5802.452508] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5802.624717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5807.981441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.152842] Lustre: lustre-OST0001: new disk, initializing [ 5808.155531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5808.161483] Lustre: 220994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5813.590975] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5814.393361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5814.428222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5814.517174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5822.052679] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5825.847732] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5826.041391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5830.042359] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5830.225303] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5833.198676] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.205172] Lustre: Skipped 5 previous similar messages [ 5838.439362] Lustre: server umount lustre-OST0001 complete [ 5845.236349] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5845.473850] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5850.363179] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5850.601438] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5879.163551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5879.402033] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.415494] Lustre: Skipped 5 previous similar messages [ 5884.406070] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5884.419144] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5884.422042] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5884.447287] Lustre: Skipped 2 previous similar messages [ 5885.401246] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5892.583371] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5894.011417] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5899.745523] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5899.754623] LustreError: Skipped 2 previous similar messages [ 5902.065460] Lustre: server umount lustre-OST0001 complete [ 5925.066264] Lustre: server umount lustre-OST0000 complete [ 5934.850631] Lustre: server umount lustre-MDT0000 complete [ 5938.294747] LustreError: 217606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824034 with bad export cookie 13955237054176464890 [ 5938.307260] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5938.311358] LustreError: 217606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5944.843793] Lustre: server umount lustre-MDT0001 complete [ 5967.284876] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:47:42 (1787824062) [ 5973.293751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.543671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5973.566125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5977.821572] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5985.453654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.580598] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5985.628350] Lustre: 226234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5989.595782] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 5996.845286] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6001.942218] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6008.485147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.753088] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6016.248986] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6024.028512] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6032.925467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.005766] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6038.632492] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6045.552597] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6047.004707] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6050.797372] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6051.019485] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6055.601248] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6055.787950] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6060.923454] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e190496c800.ost_server_uuid 50 [ 6062.150506] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e190496c800.ost_server_uuid in IDLE state after 0 sec [ 6062.943586] Lustre: Permanently deactivating lustre-OST0001 [ 6062.962257] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6066.143732] Lustre: Permanently reactivating lustre-OST0001 [ 6070.258647] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.258664] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6070.274242] LustreError: Skipped 3 previous similar messages [ 6070.298374] Lustre: Skipped 18 previous similar messages [ 6076.239698] Lustre: server umount lustre-OST0001 complete [ 6080.488118] LustreError: 228077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.500185] LustreError: 228077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 6085.780363] Lustre: server umount lustre-OST0000 complete [ 6090.723308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.733794] Lustre: Skipped 12 previous similar messages [ 6095.903730] Lustre: server umount lustre-MDT0000 complete [ 6099.352650] LustreError: 226227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824195 with bad export cookie 13955237054176466185 [ 6099.359876] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6099.361106] LustreError: 226227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6099.613852] Lustre: server umount lustre-MDT0001 complete [ 6118.029260] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:50:13 (1787824213) [ 6123.935577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.170615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6124.206382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6124.215251] Lustre: Skipped 5 previous similar messages [ 6128.162930] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6134.880085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.003125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6135.026176] Lustre: 232925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6138.949891] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6145.841138] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6150.679621] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6157.206223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6157.425119] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6157.447715] Lustre: Permanently deactivating lustre-OST0000 [ 6157.460295] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6163.483123] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6167.090243] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6169.691430] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6176.023711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6176.129837] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6182.321647] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6184.136233] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6189.498334] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6191.042466] Lustre: Permanently reactivating lustre-OST0000 [ 6191.051361] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6191.055285] Lustre: Skipped 2 previous similar messages [ 6196.786718] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6196.793419] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.813793] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6196.860666] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6196.875573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6208.206731] Lustre: server umount lustre-OST0001 complete [ 6218.257764] Lustre: server umount lustre-OST0000 complete [ 6228.436379] Lustre: server umount lustre-MDT0000 complete [ 6231.885421] LustreError: 232918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824328 with bad export cookie 13955237054176467837 [ 6231.898657] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.911600] LustreError: 232918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6232.332748] Lustre: server umount lustre-MDT0001 complete [ 6241.854910] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 6244.371914] Key type lgssc unregistered [ 6244.689867] LNet: 237802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6244.695783] LNetError: 237802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6244.714221] LNet: Removed LNI 192.168.206.148@tcp [ 6245.544092] Key type .llcrypt unregistered [ 6245.546196] Key type ._llcrypt unregistered [ 6265.087525] Key type ._llcrypt registered [ 6265.089332] Key type .llcrypt registered [ 6265.199208] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 6266.130839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6266.179781] alg: No test for adler32 (adler32-zlib) [ 6267.177174] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6267.367664] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 6269.040170] Key type lgssc registered [ 6270.227258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6278.848689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6278.881783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.351452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6284.135302] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6289.894985] LustreError: 239265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.915108] LustreError: 239265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6292.232896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.345847] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6292.366708] Lustre: 239253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6292.374210] Lustre: Permanently deactivating lustre-MDT0001 [ 6292.383775] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6292.600905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6297.012829] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6299.154143] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6303.383980] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6308.182185] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6313.961175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6314.233445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.296566] LustreError: 241107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.301458] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6319.873971] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6322.665262] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.736741] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.746036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6325.759657] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6326.677372] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6330.852801] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.868554] LustreError: 241106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6332.346738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6332.517143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6337.539585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6337.546220] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6337.627828] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6344.488442] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6348.089084] Lustre: Permanently reactivating lustre-MDT0001 [ 6348.114919] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6348.120986] Lustre: Skipped 2 previous similar messages [ 6353.496674] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6353.515731] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6353.522782] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6353.534310] LustreError: 242833:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6353.539313] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6353.552464] LustreError: 242833:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6353.567249] LustreError: 242833:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.148@tcp: cfg command failed: rc = -108 [ 6353.577898] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6353.577898] [ 6353.585559] LustreError: 239256:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6353.983167] Lustre: lustre-MDT0001: Client 6fed189f-e566-4b18-8edd-99b834de41a3 (at 192.168.206.48@tcp) reconnecting [ 6359.943902] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6361.526416] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6366.523740] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6366.701696] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6371.176540] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6371.359177] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6373.672745] Lustre: Permanently deactivating lustre-MDT0001 [ 6373.688873] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6373.693213] Lustre: Skipped 2 previous similar messages [ 6380.133052] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6394.850760] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6394.862764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6394.862991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6394.890471] Lustre: Skipped 2 previous similar messages [ 6396.639407] Lustre: server umount lustre-MDT0000 complete [ 6399.969234] LustreError: 239265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.978468] LustreError: 239265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6400.219295] LustreError: 240016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824496 with bad export cookie 7407125032250633184 [ 6400.225492] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6400.230536] LustreError: 240016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6400.745734] Lustre: server umount lustre-MDT0001 complete [ 6406.563562] Lustre: server umount lustre-OST0000 complete [ 6418.008556] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:55:13 (1787824513) [ 6429.566544] Lustre: server umount lustre-OST0001 complete [ 6437.105947] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 6446.614315] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 6481.571760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.828368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6481.842856] Lustre: Skipped 2 previous similar messages [ 6481.881229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6482.003967] Lustre: lustre-MDT0000: new disk, initializing [ 6482.096156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6482.115090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6486.925631] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6496.646713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.779342] Lustre: 247543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6496.849920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6496.854081] Lustre: Skipped 1 previous similar message [ 6496.935533] Lustre: lustre-MDT0001: new disk, initializing [ 6497.022340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.066725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6497.078686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6501.091812] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6510.991839] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6516.676276] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6524.138664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6524.415838] Lustre: lustre-OST0000: new disk, initializing [ 6524.421469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6524.433655] Lustre: 249509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6524.583302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.105229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6530.117745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6530.241763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6531.430415] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6541.393689] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6543.437278] LustreError: 249258:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6551.925653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6552.131709] Lustre: lustre-OST0001: new disk, initializing [ 6552.138676] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6552.145714] Lustre: 250624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6553.529110] LustreError: 249258:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6553.552654] LustreError: 250332:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6558.080179] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6560.861530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6560.871602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6561.003755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6563.553304] LustreError: 250332:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6567.085496] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6568.456487] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6571.495755] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6571.504081] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6571.504341] LustreError: Skipped 1 previous similar message [ 6571.505128] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6571.520373] Lustre: Skipped 2 previous similar messages [ 6571.535566] Lustre: Skipped 4 previous similar messages [ 6572.607976] Lustre: lustre-OST0001: Not available for connect from 192.168.206.48@tcp (stopping) [ 6576.590715] Lustre: server umount lustre-OST0001 complete [ 6576.625484] LustreError: 249492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6576.647504] LustreError: 249492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6597.099728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6597.101420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6597.132889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6597.135314] Lustre: Skipped 1 previous similar message [ 6602.212432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6602.224322] Lustre: Skipped 1 previous similar message [ 6607.328352] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6607.567891] Lustre: server umount lustre-OST0000 complete [ 6612.449816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.455413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6612.460407] Lustre: Skipped 2 previous similar messages [ 6612.469340] Lustre: Skipped 3 previous similar messages [ 6617.045270] Lustre: server umount lustre-MDT0000 complete [ 6617.570893] LustreError: 248708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.586040] LustreError: 248708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6620.363469] LustreError: 247536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824716 with bad export cookie 7407125032250636222 [ 6620.368800] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.393553] LustreError: 247536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6626.855526] Lustre: server umount lustre-MDT0001 complete [ 6637.259296] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 6640.251399] Key type lgssc unregistered [ 6640.638707] LNet: 252664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6640.643209] LNetError: 252664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6640.668902] LNet: Removed LNI 192.168.206.148@tcp [ 6641.431784] Key type .llcrypt unregistered [ 6641.433426] Key type ._llcrypt unregistered [ 6676.631432] Key type ._llcrypt registered [ 6676.634084] Key type .llcrypt registered [ 6676.733378] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 6677.964625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6678.010195] alg: No test for adler32 (adler32-zlib) [ 6679.007696] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6679.246925] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 6680.928158] Key type lgssc registered [ 6682.144339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6688.504797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6688.519947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.725195] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6689.743948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6689.927552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.111341] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6702.328886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.467428] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6702.506369] Lustre: 254867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6702.712945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6706.555869] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6712.457480] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6717.147159] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.003724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.219042] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6723.367781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.802491] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6731.939776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6737.681995] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.067188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6788.071527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.105550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6788.578685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6788.590899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.606590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6793.820454] Lustre: server umount lustre-OST0000 complete [ 6798.025219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6810.494531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.881216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.001444] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6818.036337] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6818.063514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6818.067742] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6818.088369] Lustre: Skipped 1 previous similar message [ 6823.794626] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6864.357047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.369955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6864.384549] Lustre: Skipped 1 previous similar message [ 6864.416029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6868.895183] Lustre: server umount lustre-OST0000 complete [ 6874.595970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.603567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6874.609059] Lustre: Skipped 1 previous similar message [ 6874.616881] Lustre: Skipped 2 previous similar messages [ 6878.519375] Lustre: server umount lustre-MDT0000 complete [ 6879.713057] LustreError: 257577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6879.728094] LustreError: 257577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6881.980471] LustreError: 256731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824978 with bad export cookie 12258049269071682855 [ 6881.991600] LustreError: 256731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6882.000395] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6882.338225] Lustre: server umount lustre-MDT0001 complete [ 6892.524908] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 6895.007123] Key type lgssc unregistered [ 6895.272753] LNet: 260305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6895.277515] LNetError: 260305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6895.293632] LNet: Removed LNI 192.168.206.148@tcp [ 6896.065887] Key type .llcrypt unregistered [ 6896.067627] Key type ._llcrypt unregistered [ 6919.584776] Key type ._llcrypt registered [ 6919.587282] Key type .llcrypt registered [ 6919.675785] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 6920.612967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6920.773178] alg: No test for adler32 (adler32-zlib) [ 6921.794268] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 6921.979258] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 6923.617265] Key type lgssc registered [ 6924.990805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6932.866391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6932.878705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.376363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.336339] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6944.737584] LustreError: 261722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6944.749566] LustreError: 261722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6946.506654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.935302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6951.297750] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6958.126377] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6963.418713] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6969.745607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.094039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.488952] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 6979.321236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6983.226484] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7005.157152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7005.163057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7005.164086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7005.170940] Lustre: Skipped 1 previous similar message [ 7010.277974] LustreError: 263544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.292964] LustreError: 263544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7010.381756] Lustre: server umount lustre-OST0000 complete [ 7015.396511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.417314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7015.423306] Lustre: Skipped 1 previous similar message [ 7020.253375] Lustre: server umount lustre-MDT0000 complete [ 7020.513617] LustreError: 261723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.530487] LustreError: 261723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7023.448214] LustreError: 261703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825119 with bad export cookie 16321228364769611341 [ 7023.455174] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7023.456729] LustreError: 261703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7023.858175] Lustre: server umount lustre-MDT0001 complete [ 7034.141900] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 7037.288800] Key type lgssc unregistered [ 7037.609961] LNet: 266028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7037.614489] LNetError: 266028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7037.633428] LNet: Removed LNI 192.168.206.148@tcp [ 7038.523362] Key type .llcrypt unregistered [ 7038.533101] Key type ._llcrypt unregistered [ 7056.503333] Key type ._llcrypt registered [ 7056.508018] Key type .llcrypt registered [ 7056.609586] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 7057.566868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7057.708731] alg: No test for adler32 (adler32-zlib) [ 7058.801434] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7059.137945] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 7060.856164] Key type lgssc registered [ 7062.058697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7069.779746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7069.812460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.222688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.574357] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7081.450993] LustreError: 267251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7081.472975] LustreError: 267251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7083.341595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7083.763023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7088.048920] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7094.798139] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7100.152277] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7106.611545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7106.964636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.844382] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7116.217313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7119.812528] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7131.622241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7131.625089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.633037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.644836] Lustre: Skipped 1 previous similar message [ 7132.503514] Lustre: server umount lustre-OST0000 complete [ 7136.738979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.763599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7136.772512] Lustre: Skipped 2 previous similar messages [ 7141.861305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7142.809619] Lustre: server umount lustre-MDT0000 complete [ 7145.985033] LustreError: 267230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825242 with bad export cookie 17093555580733735133 [ 7145.997323] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7146.278873] Lustre: server umount lustre-MDT0001 complete [ 7157.159830] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 7161.316308] Key type lgssc unregistered [ 7161.617340] LNet: 270930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7161.627271] LNetError: 270930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7161.647989] LNet: Removed LNI 192.168.206.148@tcp [ 7162.632991] Key type .llcrypt unregistered [ 7162.641404] Key type ._llcrypt unregistered [ 7188.172935] Key type ._llcrypt registered [ 7188.175984] Key type .llcrypt registered [ 7188.314623] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 7189.565917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7189.696279] alg: No test for adler32 (adler32-zlib) [ 7190.852293] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7191.075969] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 7192.744204] Key type lgssc registered [ 7193.817766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7201.621729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7201.652747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.155341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.674429] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7213.551254] LustreError: 272348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.587068] LustreError: 272348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7215.964241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.471339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7220.792964] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7227.985907] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7233.668927] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7239.944331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.338558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.587494] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7248.637436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7254.039933] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7277.537714] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7277.557382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7277.576664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7279.585701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7279.589986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7279.600663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7279.631837] Lustre: Skipped 1 previous similar message [ 7283.510363] Lustre: server umount lustre-OST0000 complete [ 7288.302784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7288.312711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.332143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7293.151151] Lustre: server umount lustre-MDT0000 complete [ 7294.951996] LustreError: 272345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.967708] LustreError: 272345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7296.889239] LustreError: 272328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825393 with bad export cookie 16580796300858921647 [ 7296.897565] LustreError: 272328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7296.901109] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7297.366826] Lustre: server umount lustre-MDT0001 complete [ 7308.205486] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 7311.233858] Key type lgssc unregistered [ 7311.638736] LNet: 276658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7311.650759] LNetError: 276658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7311.681907] LNet: Removed LNI 192.168.206.148@tcp [ 7312.511563] Key type .llcrypt unregistered [ 7312.515064] Key type ._llcrypt unregistered [ 7327.203653] Key type ._llcrypt registered [ 7327.206956] Key type .llcrypt registered [ 7327.317243] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 7328.153934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7328.237035] alg: No test for adler32 (adler32-zlib) [ 7329.265649] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7329.558555] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 7331.256233] Key type lgssc registered [ 7332.397865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7340.450965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7340.491137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.912437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.003419] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7352.296270] LustreError: 277871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7352.328107] LustreError: 277871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7354.114256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.573849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7358.559532] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7365.769745] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7371.151385] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7376.938315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.278278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.925921] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7387.527520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7390.224772] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.989770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.991546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7403.002115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.430743] Lustre: server umount lustre-OST0000 complete [ 7410.659877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7410.673063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.701956] Lustre: Skipped 1 previous similar message [ 7410.709800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7410.715096] Lustre: Skipped 1 previous similar message [ 7413.219346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.227268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7413.249915] Lustre: Skipped 1 previous similar message [ 7416.523083] Lustre: server umount lustre-MDT0000 complete [ 7418.341949] LustreError: 277876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.386099] LustreError: 277876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7420.110303] LustreError: 277838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825516 with bad export cookie 6474576775639886676 [ 7420.111644] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7420.118963] LustreError: 277838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7420.425667] Lustre: server umount lustre-MDT0001 complete [ 7431.127272] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 7434.114705] Key type lgssc unregistered [ 7434.500901] LNet: 281574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7434.509456] LNetError: 281574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7434.539078] LNet: Removed LNI 192.168.206.148@tcp [ 7435.568696] Key type .llcrypt unregistered [ 7435.575957] Key type ._llcrypt unregistered [ 7447.305851] Key type ._llcrypt registered [ 7447.313210] Key type .llcrypt registered [ 7447.471376] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7463.518715] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 7477.527752] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 7478.443508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7478.514226] alg: No test for adler32 (adler32-zlib) [ 7479.573441] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7479.825715] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 7481.522271] Key type lgssc registered [ 7482.534337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7513.658213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7513.715769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.999326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7515.056369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7515.134432] Lustre: lustre-MDT0000: new disk, initializing [ 7515.234579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7515.250514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7519.588616] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7529.560799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.697296] Lustre: 285125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7529.756975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7529.762956] Lustre: Skipped 1 previous similar message [ 7529.852465] Lustre: lustre-MDT0001: new disk, initializing [ 7529.977736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7530.020777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7530.029063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7533.925246] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7542.426438] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7547.610174] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7554.550943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.859207] Lustre: lustre-OST0000: new disk, initializing [ 7554.861645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7554.865675] Lustre: 287093:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7554.970069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.622949] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7563.859623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7563.900123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7563.995548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7570.888857] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7575.214589] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7575.446294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7579.983492] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7580.146974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7584.743093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.752938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7584.765301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7588.071594] Lustre: server umount lustre-OST0000 complete [ 7594.978729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.983426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7594.994738] Lustre: Skipped 2 previous similar messages [ 7595.006868] Lustre: Skipped 2 previous similar messages [ 7597.730606] Lustre: server umount lustre-MDT0000 complete [ 7601.378654] LustreError: 285118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825697 with bad export cookie 11141930597674270561 [ 7601.391470] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.405352] LustreError: 285118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7601.680460] Lustre: server umount lustre-MDT0001 complete [ 7609.658107] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:15:04 (1787825704) [ 7615.168600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.583731] LustreError: 289505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.603130] LustreError: 289505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7615.658789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7620.376316] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7621.090100] LustreError: 289506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.189892] LustreError: 289505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.180570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.632447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7632.587878] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7638.840531] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7643.161779] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7648.907024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7649.220344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.391987] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7661.159384] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7670.758357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7670.762659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.764949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7670.765366] LustreError: Skipped 1 previous similar message [ 7670.788591] Lustre: Skipped 1 previous similar message [ 7675.877265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.880929] Lustre: Skipped 2 previous similar messages [ 7676.735981] Lustre: server umount lustre-OST0000 complete [ 7680.995346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7681.004584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7681.019839] Lustre: Skipped 1 previous similar message [ 7686.835431] Lustre: server umount lustre-MDT0000 complete [ 7690.603228] LustreError: 289485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825786 with bad export cookie 11141930597674271443 [ 7690.633370] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7691.017810] Lustre: server umount lustre-MDT0001 complete [ 7701.306233] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 7704.074938] Key type lgssc unregistered [ 7704.443488] LNet: 293041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7704.447987] LNetError: 293041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7704.458757] LNet: Removed LNI 192.168.206.148@tcp [ 7705.032131] Key type .llcrypt unregistered [ 7705.035425] Key type ._llcrypt unregistered [ 7733.093659] Key type ._llcrypt registered [ 7733.098449] Key type .llcrypt registered [ 7733.211474] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 7748.579835] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 7749.772423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7750.072198] alg: No test for adler32 (adler32-zlib) [ 7751.210493] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 7751.485163] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 7753.192184] Key type lgssc registered [ 7754.241988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7796.001102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7796.035293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.320559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7797.358979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7797.436468] Lustre: lustre-MDT0000: new disk, initializing [ 7797.530324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.554683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7801.357555] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7812.796286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.929291] Lustre: 296872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7813.001908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7813.005953] Lustre: Skipped 1 previous similar message [ 7813.107275] Lustre: lustre-MDT0001: new disk, initializing [ 7813.235351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7813.285274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7813.296154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7818.260844] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7826.880354] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7830.898147] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7837.551972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7837.777662] Lustre: lustre-OST2710: new disk, initializing [ 7837.782363] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7837.786823] Lustre: 298840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7837.876188] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7843.921062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7843.932270] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7844.065619] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7844.102714] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7854.254701] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7860.737826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7860.942402] Lustre: lustre-OST03e8: new disk, initializing [ 7860.945459] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7860.951693] Lustre: 299906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7861.042856] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7866.986418] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7870.089564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7870.097874] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7870.191788] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7874.421269] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7880.719307] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7880.947432] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7885.577940] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7885.795987] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7890.435756] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7890.678917] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7895.002634] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7895.245767] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7900.129183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7900.137383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7900.169193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7901.158335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.162196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7901.177276] Lustre: Skipped 1 previous similar message [ 7901.188271] Lustre: Skipped 3 previous similar messages [ 7901.580700] Lustre: server umount lustre-MDT0000 complete [ 7904.928862] LustreError: 296863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826001 with bad export cookie 3532050288197424619 [ 7904.937330] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7904.940990] LustreError: 296863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7905.434339] Lustre: server umount lustre-MDT0001 complete [ 7919.316824] Lustre: server umount lustre-OST2710 complete [ 7922.656195] Lustre: 294670:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787826002/real 1787826002] req@000000001270f916 x1874671275408000/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787826018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7922.692527] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.710204] Lustre: Skipped 1 previous similar message [ 7923.323849] Lustre: server umount lustre-OST03e8 complete [ 7939.761596] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 7948.963884] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 7984.097694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.379459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7984.412265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7984.492186] Lustre: lustre-MDT0000: new disk, initializing [ 7984.584642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.612784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7989.645516] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 7999.868512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.979922] Lustre: 305149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8000.023457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8000.033271] Lustre: Skipped 1 previous similar message [ 8000.123260] Lustre: lustre-MDT0001: new disk, initializing [ 8000.265088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8000.336482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8000.351439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8005.194941] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8013.865933] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8019.354565] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.957356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8026.331568] Lustre: lustre-OST0000: new disk, initializing [ 8026.336264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8026.343031] Lustre: 307118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8026.426978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.327246] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8033.839911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8033.845920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8033.953791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8041.083419] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8046.337241] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8046.542547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8050.768644] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8050.955436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8054.242045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8054.250668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.263023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8058.742493] Lustre: server umount lustre-OST0000 complete [ 8064.996252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8064.999927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8065.023458] Lustre: Skipped 2 previous similar messages [ 8065.029037] Lustre: Skipped 1 previous similar message [ 8069.078993] Lustre: server umount lustre-MDT0000 complete [ 8073.038325] LustreError: 305142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826169 with bad export cookie 3532050288197426299 [ 8073.041678] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8073.049334] LustreError: 305142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8073.465106] Lustre: server umount lustre-MDT0001 complete [ 8082.514689] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 06:22:57 (1787826177) [ 8083.796202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8085.847557] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 06:23:00 (1787826180) [ 8090.070386] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 8116.971549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.299173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8117.323851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8117.491127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.176556] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8131.000247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.099829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8131.132960] Lustre: 310977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8135.899976] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8157.224805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8157.383040] Lustre: Denying initial registration attempt from nid 192.168.206.148@tcp, specified as failover [ 8157.396680] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8157.406710] LustreError: 312432:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 8157.412444] LustreError: 312432:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8157.417240] LustreError: 312432:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8157.470547] Lustre: server umount lustre-OST0000 complete [ 8157.473921] LustreError: 312432:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8163.810810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8163.811679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8163.836535] Lustre: Skipped 1 previous similar message [ 8163.839866] Lustre: Skipped 1 previous similar message [ 8171.724805] Lustre: server umount lustre-MDT0000 complete [ 8174.053604] LustreError: 310990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.078432] LustreError: 310990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8175.331122] LustreError: 311721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826271 with bad export cookie 3532050288197427181 [ 8175.335456] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8175.340163] LustreError: 311721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8175.768298] Lustre: server umount lustre-MDT0001 complete [ 8187.170617] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 8189.937156] Key type lgssc unregistered [ 8190.233743] LNet: 313557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8190.241225] LNetError: 313557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8190.252576] LNet: Removed LNI 192.168.206.148@tcp [ 8191.040702] Key type .llcrypt unregistered [ 8191.042716] Key type ._llcrypt unregistered [ 8201.106933] Key type ._llcrypt registered [ 8201.109220] Key type .llcrypt registered [ 8201.239181] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 8202.106115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8202.290121] alg: No test for adler32 (adler32-zlib) [ 8203.284350] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 8203.616687] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 8205.320261] Key type lgssc registered [ 8206.322350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8255.337525] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 8265.100534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8265.116438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8266.382949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8266.418940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8266.651068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.460172] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8279.758331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.884238] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8279.916672] Lustre: 316321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8280.144455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8284.400323] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8291.160805] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8297.271910] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8317.870048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8318.151662] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8318.382209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8324.410866] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8332.191372] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8337.889325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8337.898303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.916967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8338.402406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8338.419133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8338.423821] Lustre: Skipped 1 previous similar message [ 8340.574923] Lustre: server umount lustre-OST0000 complete [ 8346.593604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8346.605265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.657370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8348.641206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.806960] Lustre: server umount lustre-MDT0000 complete [ 8354.388977] LustreError: 316315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826450 with bad export cookie 684834254377400860 [ 8354.406517] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.407782] LustreError: 316315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8354.791432] Lustre: server umount lustre-MDT0001 complete [ 8366.206867] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 8369.235956] Key type lgssc unregistered [ 8369.597678] LNet: 319898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8369.602883] LNetError: 319898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8369.635787] LNet: Removed LNI 192.168.206.148@tcp [ 8370.410700] Key type .llcrypt unregistered [ 8370.413765] Key type ._llcrypt unregistered [ 8397.435790] Key type ._llcrypt registered [ 8397.440378] Key type .llcrypt registered [ 8397.575466] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 8398.770637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8398.993962] alg: No test for adler32 (adler32-zlib) [ 8400.105611] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 8400.476551] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 8402.312191] Key type lgssc registered [ 8403.915175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8412.602764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8412.625244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.138091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.056517] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8424.426549] LustreError: 321316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.458450] LustreError: 321316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8428.529195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.985158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8433.951659] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8441.617427] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8447.436898] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8456.110173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.434470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.275398] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8471.230819] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8489.553316] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.48@tcp (stopping) [ 8490.480963] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8490.485635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8490.487029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8490.489111] Lustre: Skipped 1 previous similar message [ 8492.273443] Lustre: server umount lustre-MDT0000 complete [ 8494.644938] LustreError: 321311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8496.521774] LustreError: 323927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826592 with bad export cookie 8746040978627553658 [ 8496.523911] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.533368] LustreError: 323927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8499.735681] LustreError: 321311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8499.755607] LustreError: 321311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8499.770491] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.48@tcp (stopping) [ 8499.781121] Lustre: Skipped 2 previous similar messages [ 8500.708285] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.722629] Lustre: Skipped 1 previous similar message [ 8503.308256] Lustre: server umount lustre-MDT0001 complete [ 8509.924686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8518.611619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.184833] LustreError: 320627:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ff645c6d x1874671955911936/t0(0) o250->MGC192.168.206.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8521.599047] LustreError: 323149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.633170] LustreError: 323149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8521.709305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.780184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8526.141629] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8526.824522] LustreError: 323150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.842595] LustreError: 323150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8530.928478] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8534.608888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.958962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8540.154572] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8540.579751] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8549.148522] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8550.761643] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8555.044882] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8556.347222] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8560.614445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8560.617910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8560.625065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8560.637959] Lustre: Skipped 1 previous similar message [ 8560.651362] Lustre: Skipped 2 previous similar messages [ 8565.730214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8565.738261] Lustre: Skipped 2 previous similar messages [ 8566.377875] Lustre: server umount lustre-OST0000 complete [ 8570.849882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.877627] Lustre: Skipped 1 previous similar message [ 8575.977236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8575.991039] Lustre: Skipped 3 previous similar messages [ 8577.110975] Lustre: server umount lustre-MDT0000 complete [ 8580.605651] LustreError: 321298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826676 with bad export cookie 8746040978627561295 [ 8580.611871] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8580.617531] LustreError: 321298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8580.967991] Lustre: server umount lustre-MDT0001 complete [ 8591.373794] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 8594.276868] Key type lgssc unregistered [ 8594.579684] LNet: 327611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8594.587514] LNetError: 327611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8594.611962] LNet: Removed LNI 192.168.206.148@tcp [ 8595.495646] Key type .llcrypt unregistered [ 8595.502187] Key type ._llcrypt unregistered [ 8617.945298] Key type ._llcrypt registered [ 8617.947199] Key type .llcrypt registered [ 8618.072522] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 8618.969856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8619.032605] alg: No test for adler32 (adler32-zlib) [ 8620.062485] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 8620.291568] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 8621.944602] Key type lgssc registered [ 8622.947728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8630.199761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8630.223707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.690192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.796447] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8641.001162] LustreError: 329025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8641.014062] LustreError: 329025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8643.028437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.412865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8647.853269] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8653.744624] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8658.764224] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8661.985521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8661.990693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8661.998242] Lustre: Skipped 1 previous similar message [ 8662.019346] Lustre: Skipped 1 previous similar message [ 8667.105956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8667.116951] Lustre: Skipped 1 previous similar message [ 8668.132489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8668.143302] Lustre: Skipped 1 previous similar message [ 8673.249828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8673.253400] Lustre: Skipped 1 previous similar message [ 8676.322624] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8676.568409] Lustre: server umount lustre-MDT0000 complete [ 8678.375933] LustreError: 329026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8678.395592] LustreError: 329026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8680.375285] LustreError: 329007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826776 with bad export cookie 13069403341779149068 [ 8680.376152] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8680.381714] LustreError: 329007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8680.797393] Lustre: server umount lustre-MDT0001 complete [ 8692.720499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.949571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8692.985785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8693.205666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.072628] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8704.429920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.566332] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8704.603464] Lustre: 331614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8704.859510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8709.463757] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8716.792913] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8722.442518] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8731.836661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.013582] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8732.023190] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8732.041737] LustreError: 333498:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 8732.053500] LustreError: 333498:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8732.069510] LustreError: 333498:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8732.172868] Lustre: server umount lustre-OST0000 complete [ 8732.174855] LustreError: 333498:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8741.303055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8741.491728] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8741.665314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.123883] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8751.275289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8754.064192] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8760.648194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8760.714977] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8760.774771] Lustre: lustre-OST0001: new disk, initializing [ 8760.777907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8760.783460] Lustre: 335038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8760.873161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8766.058148] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8768.594305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8768.612995] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8768.662404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8772.318576] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8779.245414] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.245863] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.246867] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8779.266100] Lustre: Skipped 1 previous similar message [ 8779.269249] LustreError: Skipped 1 previous similar message [ 8781.165732] Lustre: server umount lustre-OST0001 complete [ 8784.362949] LustreError: 333963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.381181] LustreError: 333963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8786.913118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8786.917272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.434389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8787.438826] Lustre: Skipped 2 previous similar messages [ 8789.490598] LustreError: 333965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8789.530316] LustreError: 333965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8790.764105] Lustre: server umount lustre-OST0000 complete [ 8797.159487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8797.170622] LustreError: Skipped 1 previous similar message [ 8797.173614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8797.189260] Lustre: Skipped 1 previous similar message [ 8801.902295] Lustre: server umount lustre-MDT0000 complete [ 8804.838074] LustreError: 331625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.857955] LustreError: 331625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8805.494721] LustreError: 332354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826901 with bad export cookie 13069403341779149621 [ 8805.501241] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8805.504913] LustreError: 332354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8805.724944] Lustre: server umount lustre-MDT0001 complete [ 8815.207908] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 8817.551128] Key type lgssc unregistered [ 8817.832460] LNet: 337000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8817.839723] LNetError: 337000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8818.854501] LNet: Removed LNI 192.168.206.148@tcp [ 8819.641437] Key type .llcrypt unregistered [ 8819.642808] Key type ._llcrypt unregistered [ 8884.404387] Key type ._llcrypt registered [ 8884.408020] Key type .llcrypt registered [ 8884.505905] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 8898.500625] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 8899.541403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8899.640368] alg: No test for adler32 (adler32-zlib) [ 8900.651720] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 8900.937445] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 8902.627334] Key type lgssc registered [ 8904.021405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8935.586539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8935.602392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.876207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8936.953669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8937.050612] Lustre: lustre-MDT0000: new disk, initializing [ 8937.130819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.151982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8941.122492] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8950.677381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.793202] Lustre: 341933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8950.873105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8950.875508] Lustre: Skipped 1 previous similar message [ 8950.941985] Lustre: lustre-MDT0001: new disk, initializing [ 8951.025713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8951.052375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8951.059544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8955.015124] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8965.705355] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8972.494231] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8979.870807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.129759] Lustre: lustre-OST0000: new disk, initializing [ 8980.133166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8980.138818] Lustre: 343905:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8980.255520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8985.781940] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 8986.184853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8986.201556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8986.352166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8994.195268] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8998.238752] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8998.462821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9002.402912] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9002.596763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9007.083440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9007.084272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.088703] LustreError: Skipped 1 previous similar message [ 9007.092120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9007.097208] Lustre: Skipped 1 previous similar message [ 9010.451522] Lustre: server umount lustre-OST0000 complete [ 9017.321927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.326665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.335034] Lustre: Skipped 1 previous similar message [ 9017.342781] Lustre: Skipped 2 previous similar messages [ 9022.433983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9022.440180] Lustre: Skipped 1 previous similar message [ 9027.557165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9027.560615] Lustre: Skipped 1 previous similar message [ 9028.066675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9028.264561] Lustre: server umount lustre-MDT0000 complete [ 9031.946609] LustreError: 343903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827128 with bad export cookie 8243061012568595463 [ 9031.955674] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9031.969491] LustreError: 343903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9032.294885] Lustre: server umount lustre-MDT0001 complete [ 9039.963040] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:38:55 (1787827135) [ 9046.310304] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:39:01 (1787827141) [ 9053.325416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.796634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.342152] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9058.799847] LustreError: 346662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9058.828024] LustreError: 346662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9062.884795] LustreError: 346661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.043719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.690296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9072.032188] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9079.827451] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9086.053272] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9093.003235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9093.527495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9100.790668] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9108.520401] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9120.331621] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9122.168564] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9123.757453] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9125.343506] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9127.015104] Lustre: Failing over lustre-MDT0000 [ 9127.268531] Lustre: server umount lustre-MDT0000 complete [ 9127.398032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9127.412523] LustreError: 346658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9127.425284] LustreError: 346658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9131.573367] LustreError: 348741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9131.594903] LustreError: 348741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9136.663973] LustreError: 346657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9136.686789] LustreError: 346657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9145.189951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.313191] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9145.473561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.514767] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9146.905332] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9149.693626] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9150.974646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9151.018881] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9151.060526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9158.957172] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9160.198480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9163.818122] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9165.687697] Lustre: Failing over lustre-MDT0000 [ 9165.957834] Lustre: server umount lustre-MDT0000 complete [ 9166.308117] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.311702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9166.321303] Lustre: Skipped 2 previous similar messages [ 9166.324091] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.354127] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9182.688173] Lustre: 340160:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827262/real 1787827262] req@00000000a7ec1273 x1874672481293952/t0(0) o400->MGC192.168.206.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787827278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9182.719448] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.744230] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9182.775050] LustreError: 346656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 9184.632152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.905594] LustreError: 340157:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000040c856cf x1874672481299328/t0(0) o250->MGC192.168.206.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9192.227243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.277975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9192.998376] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9196.784739] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9197.568141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9197.579393] Lustre: Skipped 2 previous similar messages [ 9197.661378] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9197.736995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9207.072627] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9208.963324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9218.026554] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.027958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9218.046642] Lustre: Skipped 4 previous similar messages [ 9218.059715] Lustre: Skipped 2 previous similar messages [ 9223.139953] LustreError: 346662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9223.174981] LustreError: 346662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9223.568877] Lustre: server umount lustre-MDT0000 complete [ 9227.656762] LustreError: 346642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827323 with bad export cookie 8243061012568598165 [ 9227.677027] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9227.679659] LustreError: 346642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9227.930368] Lustre: server umount lustre-MDT0001 complete [ 9241.290138] Lustre: server umount lustre-OST0000 complete [ 9256.474570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.861587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9261.589917] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9269.917630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9274.679737] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9281.118575] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9286.633213] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9293.292291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9299.683385] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9301.894093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9305.537706] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9311.202439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9311.213639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9311.238298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.213890] Lustre: server umount lustre-OST0000 complete [ 9321.440917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9321.448129] LustreError: Skipped 1 previous similar message [ 9324.889217] Lustre: server umount lustre-MDT0000 complete [ 9327.585519] LustreError: 353019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9327.602181] LustreError: 353019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9328.761943] LustreError: 353004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827424 with bad export cookie 8243061012568598914 [ 9328.772554] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9328.787768] LustreError: 353004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9329.315630] Lustre: server umount lustre-MDT0001 complete [ 9338.939366] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 9341.738392] Key type lgssc unregistered [ 9342.180417] LNet: 356556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9342.191842] LNetError: 356556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9342.209367] LNet: Removed LNI 192.168.206.148@tcp [ 9343.244097] Key type .llcrypt unregistered [ 9343.245774] Key type ._llcrypt unregistered [ 9366.994473] Key type ._llcrypt registered [ 9366.997443] Key type .llcrypt registered [ 9367.100172] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 9367.892907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9367.902458] alg: No test for adler32 (adler32-zlib) [ 9368.920900] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 9369.147823] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 9370.848164] Key type lgssc registered [ 9371.805958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9378.234658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9378.248611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.626712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9383.581864] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9389.030592] LustreError: 358075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.050846] LustreError: 358075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9390.662572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.968515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9394.885571] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9401.892290] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9407.404462] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9413.674674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9414.024208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.542718] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9424.256853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9426.724708] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9439.733815] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.747415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.758941] Lustre: Skipped 2 previous similar messages [ 9439.763648] Lustre: Skipped 1 previous similar message [ 9440.216557] Lustre: server umount lustre-MDT0000 complete [ 9444.064394] LustreError: 358056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827540 with bad export cookie 17874252053514613364 [ 9444.076499] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9444.077804] LustreError: 358056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9444.478623] Lustre: server umount lustre-MDT0001 complete [ 9458.855763] Lustre: server umount lustre-OST0000 complete [ 9471.047200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.449604] LustreError: 361786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9471.512869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9475.926595] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9476.582804] LustreError: 361787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.714771] LustreError: 361786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.334606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.669318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9489.350936] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9496.565434] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9502.175558] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9508.295020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.722400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9514.612924] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9515.914622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9522.875880] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9535.974073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9535.986359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9536.003794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9536.016266] Lustre: Skipped 1 previous similar message [ 9540.848124] Lustre: server umount lustre-MDT0000 complete [ 9541.610784] LustreError: 361786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.636623] LustreError: 361786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9544.141124] LustreError: 361766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827640 with bad export cookie 17874252053514614477 [ 9544.150172] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9544.162719] LustreError: 361766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9544.459128] Lustre: server umount lustre-MDT0001 complete [ 9558.339970] Lustre: server umount lustre-OST0000 complete [ 9572.206371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9572.692169] LustreError: 365694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.752783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.803174] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9584.447482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.716372] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9594.834647] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9600.478448] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9607.808192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9608.104928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9608.110957] Lustre: Skipped 1 previous similar message [ 9614.177654] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9617.342813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9621.408388] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9627.618220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9627.623880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9627.641956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9627.641965] Lustre: Skipped 3 previous similar messages [ 9627.685853] Lustre: Skipped 3 previous similar messages [ 9632.557103] Lustre: server umount lustre-OST0000 complete [ 9636.322880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9636.334717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9636.360283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9636.368453] Lustre: Skipped 1 previous similar message [ 9642.419287] Lustre: server umount lustre-MDT0000 complete [ 9642.977684] LustreError: 366765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9643.005171] LustreError: 366765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9646.581200] LustreError: 365675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827742 with bad export cookie 17874252053514615499 [ 9646.584933] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9646.597367] LustreError: 365675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9647.096768] Lustre: server umount lustre-MDT0001 complete [ 9658.415628] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 9661.517550] Key type lgssc unregistered [ 9661.881946] LNet: 369227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9661.904890] LNetError: 369227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9661.937067] LNet: Removed LNI 192.168.206.148@tcp [ 9662.904086] Key type .llcrypt unregistered [ 9662.906349] Key type ._llcrypt unregistered [ 9687.652073] Key type ._llcrypt registered [ 9687.653477] Key type .llcrypt registered [ 9687.730187] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 9688.526152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9688.563290] alg: No test for adler32 (adler32-zlib) [ 9689.765989] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 9690.013530] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 9691.667525] Key type lgssc registered [ 9693.159289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9701.168552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9701.184284] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.188975] LustreError: 370686:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9701.205846] LustreError: 370686:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9701.216520] LustreError: 370686:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9701.226650] LustreError: 370686:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9701.234392] LustreError: 370686:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9711.607416] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9711.626937] LustreError: 371083:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9711.640972] LustreError: 371083:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9711.647621] LustreError: 371083:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9711.654443] LustreError: 371083:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9711.661124] LustreError: 371083:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9731.174551] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [ 9733.507821] Key type lgssc unregistered [ 9733.786958] LNet: 372089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9733.794986] LNetError: 372089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9733.822592] LNet: Removed LNI 192.168.206.148@tcp [ 9734.532386] Key type .llcrypt unregistered [ 9734.535474] Key type ._llcrypt unregistered [ 9750.134731] Key type ._llcrypt registered [ 9750.140978] Key type .llcrypt registered [ 9750.278641] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [ 9768.104661] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [ 9768.921892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9768.928728] alg: No test for adler32 (adler32-zlib) [ 9769.961423] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [ 9770.140673] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [ 9771.784379] Key type lgssc registered [ 9773.374571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9808.016975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9808.037813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.396587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9809.428974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9809.554685] Lustre: lustre-MDT0000: new disk, initializing [ 9809.605269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.626979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9814.438449] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9824.558821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.678171] Lustre: 375248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9824.719526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9824.722256] Lustre: Skipped 1 previous similar message [ 9824.784351] Lustre: lustre-MDT0001: new disk, initializing [ 9824.872254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9824.905266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9824.914365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9829.105897] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9838.342117] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9843.746144] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9849.813527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9850.071903] Lustre: lustre-OST0000: new disk, initializing [ 9850.074551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9850.079367] Lustre: 377214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9850.152838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9855.770192] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9859.167435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9859.184076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9859.318724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9864.703513] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9869.813197] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9870.027913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9874.810458] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9875.120921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9880.035201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9880.043398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9880.045780] LustreError: Skipped 1 previous similar message [ 9880.046438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9880.077380] Lustre: Skipped 1 previous similar message [ 9883.402797] Lustre: server umount lustre-OST0000 complete [ 9890.286310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9890.288371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9890.310351] Lustre: Skipped 1 previous similar message [ 9890.328288] Lustre: Skipped 2 previous similar messages [ 9893.435631] Lustre: server umount lustre-MDT0000 complete [ 9897.168485] LustreError: 375240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827993 with bad export cookie 11629494277368994688 [ 9897.177466] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9897.180676] LustreError: 375240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9897.689102] Lustre: server umount lustre-MDT0001 complete [ 9905.762889] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9907.446814] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:53:22 (1787828002) [ 9913.938509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9914.442905] LustreError: 379672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.461366] LustreError: 379672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9914.542169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.869855] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9919.969800] LustreError: 379673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.072088] LustreError: 379672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.718576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.143697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9932.912839] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9940.326872] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9945.545629] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9952.266404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9952.598526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.699126] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9966.926482] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9973.307180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.553602] Lustre: lustre-OST0001: new disk, initializing [ 9973.560472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9973.567302] Lustre: 382511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9973.663261] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9979.718922] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [ 9980.606852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9980.620935] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9980.800214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9989.068678] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9995.830516] Lustre: lustre-OST0001: Not available for connect from 192.168.206.48@tcp (stopping) [ 9996.275703] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9996.289786] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.002909] Lustre: server umount lustre-OST0001 complete [10000.933747] LustreError: 381497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.504107] LustreError: 381497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.528809] LustreError: 381497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10016.236839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10016.240946] LustreError: Skipped 1 previous similar message [10016.249172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10016.262460] Lustre: Skipped 1 previous similar message [10016.269315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10016.279429] Lustre: Skipped 2 previous similar messages [10016.745942] LustreError: 381496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10016.761763] LustreError: 381496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10021.858150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10021.873182] Lustre: Skipped 2 previous similar messages [10022.157163] Lustre: server umount lustre-OST0000 complete [10026.979184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10027.011198] Lustre: Skipped 1 previous similar message [10032.097277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10032.104060] Lustre: Skipped 3 previous similar messages [10032.703864] Lustre: server umount lustre-MDT0000 complete [10036.797043] LustreError: 379655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828132 with bad export cookie 11629494277368995570 [10036.806693] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10036.820960] LustreError: 379655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10043.320664] Lustre: server umount lustre-MDT0001 complete [10055.132594] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [10058.143928] Key type lgssc unregistered [10058.488241] LNet: 384500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10058.496513] LNetError: 384500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10058.514800] LNet: Removed LNI 192.168.206.148@tcp [10059.386560] Key type .llcrypt unregistered [10059.396266] Key type ._llcrypt unregistered [10105.536966] Key type ._llcrypt registered [10105.538879] Key type .llcrypt registered [10105.619834] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [10106.926499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10106.998326] alg: No test for adler32 (adler32-zlib) [10108.148544] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10108.378740] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [10110.128155] Key type lgssc registered [10111.458504] Lustre: Echo OBD driver; http://www.lustre.org/ [10119.191865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10119.210943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.502951] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10120.517906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10120.635255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10125.324884] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10133.682611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.913187] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10133.969460] Lustre: 387297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10134.271358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10139.202236] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10146.602897] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10152.458541] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10154.978068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10154.993282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10155.010705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.074030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10159.076324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.094824] Lustre: Skipped 1 previous similar message [10160.743427] Lustre: server umount lustre-MDT0000 complete [10164.329884] LustreError: 387292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828260 with bad export cookie 6778008215356958703 [10164.330620] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10164.338248] LustreError: 387292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10164.603026] Lustre: server umount lustre-MDT0001 complete [10172.434553] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:57:47 (1787828267) [10177.660406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.098659] LustreError: 389811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.116451] LustreError: 389811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10178.140648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10181.742870] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10183.140377] LustreError: 389812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.276362] LustreError: 389811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.303321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.622037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10193.252846] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10200.399753] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10205.864589] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10212.191018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10212.424475] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10212.600098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10218.099568] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10223.801161] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10231.260245] Lustre: Permanently deactivating lustre-OST0000 [10231.282977] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10237.477152] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10240.111665] LustreError: 392798:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10240.125622] LustreError: 392798:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10240.148420] LustreError: 392798:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10243.156904] LustreError: 392846:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10243.164211] LustreError: 392846:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10250.874339] Lustre: server umount lustre-OST0000 complete [10253.644037] LustreError: 393096:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10253.650671] LustreError: 393096:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10256.357496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10256.362621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10256.382077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10261.988396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10261.996982] Lustre: Skipped 2 previous similar messages [10265.173717] Lustre: server umount lustre-MDT0000 complete [10267.105694] LustreError: 389812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.133290] LustreError: 389812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10268.338883] LustreError: 389792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828364 with bad export cookie 6778008215356959284 [10268.348225] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10268.350268] LustreError: 389792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10278.050664] Lustre: server umount lustre-MDT0001 complete [10285.927686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.831767] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10292.229389] LustreError: 394253:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10297.379495] Lustre: 394592:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.148@tcp [10299.272875] Lustre: server umount MGS complete [10309.701721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.154235] LustreError: 395317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.176982] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10310.181136] Lustre: Skipped 1 previous similar message [10310.241257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10313.647551] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10320.369986] LustreError: 395318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10320.390144] LustreError: 395318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10320.747410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.012533] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10325.030653] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10330.999984] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10335.682858] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10337.282751] Lustre: Permanently reactivating lustre-OST0000 [10337.306118] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10337.310964] Lustre: Skipped 2 previous similar messages [10366.445435] LustreError: 397155:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10374.322864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10375.057263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10375.068251] Lustre: Skipped 1 previous similar message [10381.470555] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10388.693807] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10395.688927] Lustre: server umount lustre-OST0000 complete [10401.762161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10401.768602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10401.778314] Lustre: Skipped 2 previous similar messages [10401.790185] Lustre: Skipped 1 previous similar message [10409.163526] Lustre: server umount lustre-MDT0000 complete [10412.000709] Lustre: 386621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828492/real 1787828492] req@000000008e5cf7d9 x1874673746932864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787828508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10412.003554] LustreError: 397048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.003971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.035564] Lustre: 386621:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10412.096510] LustreError: 397048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10412.800734] LustreError: 396891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828508 with bad export cookie 6778008215356960334 [10412.810435] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10412.813403] LustreError: 396891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10413.144653] Lustre: server umount lustre-MDT0001 complete [10423.868924] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [10426.727603] Key type lgssc unregistered [10427.042129] LNet: 399792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10427.054486] LNetError: 399792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10427.087299] LNet: Removed LNI 192.168.206.148@tcp [10427.884165] Key type .llcrypt unregistered [10427.889319] Key type ._llcrypt unregistered [10444.113421] Key type ._llcrypt registered [10444.116854] Key type .llcrypt registered [10444.242912] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [10459.026656] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [10460.037300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10460.175419] alg: No test for adler32 (adler32-zlib) [10461.232611] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10461.487726] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [10463.168157] Key type lgssc registered [10464.334379] Lustre: Echo OBD driver; http://www.lustre.org/ [10497.420241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10497.461167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10498.722688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10498.763866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10498.883486] Lustre: lustre-MDT0000: new disk, initializing [10498.972497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10499.000716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10503.529149] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10513.471016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.674580] Lustre: 402949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10513.794883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10513.800253] Lustre: Skipped 1 previous similar message [10513.911788] Lustre: lustre-MDT0001: new disk, initializing [10514.024741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10514.069494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10514.082505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10518.993120] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10527.882393] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10533.123350] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10538.934883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10539.185706] Lustre: lustre-OST0000: new disk, initializing [10539.189228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10539.197658] Lustre: 404917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10539.307304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10544.909974] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10546.760657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10546.770420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10546.852172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10553.319842] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10557.200926] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10557.415765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10562.261810] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10562.473049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10567.653929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10567.658959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.668432] LustreError: Skipped 1 previous similar message [10567.671223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10567.685529] Lustre: Skipped 1 previous similar message [10567.697272] Lustre: Skipped 1 previous similar message [10570.867314] Lustre: server umount lustre-OST0000 complete [10575.330035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10575.338912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10575.353665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10577.890161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10577.897049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.899352] Lustre: server umount lustre-MDT0000 complete [10584.231899] LustreError: 402940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828680 with bad export cookie 802907143204512019 [10584.242130] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10584.244629] LustreError: 402940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10584.644358] Lustre: server umount lustre-MDT0001 complete [10592.425300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10594.062577] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:04:49 (1787828689) [10600.111815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.539582] LustreError: 407376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.564162] LustreError: 407376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10600.605859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10604.367832] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10606.055177] LustreError: 407377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.161027] LustreError: 407376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.298303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.768111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10618.760450] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10626.199318] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10631.999734] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10639.160263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.447260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10645.207788] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10652.259740] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10654.077115] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10656.589336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10668.513620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10668.531415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10668.539079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10668.548787] Lustre: Skipped 1 previous similar message [10668.563624] Lustre: Skipped 2 previous similar messages [10673.486759] Lustre: server umount lustre-OST0000 complete [10678.756297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.777672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10678.787559] Lustre: Skipped 1 previous similar message [10683.007760] Lustre: server umount lustre-MDT0000 complete [10683.876148] LustreError: 407373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.906911] LustreError: 407373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10686.365586] LustreError: 407357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828782 with bad export cookie 802907143204512901 [10686.381540] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10686.622939] Lustre: server umount lustre-MDT0001 complete [10696.458778] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [10699.071694] Key type lgssc unregistered [10699.316930] LNet: 411105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10699.337977] LNetError: 411105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10699.359799] LNet: Removed LNI 192.168.206.148@tcp [10700.128513] Key type .llcrypt unregistered [10700.131243] Key type ._llcrypt unregistered [10719.618320] Key type ._llcrypt registered [10719.619657] Key type .llcrypt registered [10719.714154] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [10720.051274] Key type .llcrypt unregistered [10720.053929] Key type ._llcrypt unregistered [10734.232440] Key type ._llcrypt registered [10734.235412] Key type .llcrypt registered [10734.360111] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [10735.040134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10735.224220] alg: No test for adler32 (adler32-zlib) [10736.335655] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10736.533054] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [10738.168182] Key type lgssc registered [10739.172808] Lustre: Echo OBD driver; http://www.lustre.org/ [10747.265721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10747.300645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10748.686377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10752.556646] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10758.114821] LustreError: 413300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10758.128402] LustreError: 413300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10760.426067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10760.682734] LustreError: 413301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.705517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10765.796410] LustreError: 413297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.827648] LustreError: 413297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10766.518087] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10770.924266] LustreError: 413296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.945427] LustreError: 413296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10772.974506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10773.857510] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10778.084653] LustreError: 413297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10778.112660] LustreError: 413297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10780.520379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.079245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10785.733504] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10796.415652] Lustre: server umount lustre-OST0000 complete [10801.639989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10801.645570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10801.661188] Lustre: Skipped 1 previous similar message [10801.677367] Lustre: Skipped 1 previous similar message [10806.755255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10806.765420] Lustre: Skipped 1 previous similar message [10810.928480] Lustre: server umount lustre-MDT0000 complete [10811.872099] Lustre: 412613:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787828892/real 1787828892] req@00000000d6dbaf47 x1874674405371008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787828908 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10811.877063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10811.882872] LustreError: 415037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10811.890929] Lustre: 412613:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10814.395130] LustreError: 415697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828910 with bad export cookie 14709763701549566991 [10814.403035] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10814.424633] LustreError: 415697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10814.743621] Lustre: server umount lustre-MDT0001 complete [10824.328778] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [10826.998292] Key type lgssc unregistered [10827.267688] LNet: 416450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10827.274623] LNetError: 416450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10827.298029] LNet: Removed LNI 192.168.206.148@tcp [10828.169923] Key type .llcrypt unregistered [10828.174563] Key type ._llcrypt unregistered [10852.565247] Key type ._llcrypt registered [10852.567312] Key type .llcrypt registered [10852.669862] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [10853.626981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10853.866061] alg: No test for adler32 (adler32-zlib) [10854.948981] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [10855.230712] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [10856.896163] Key type lgssc registered [10857.949559] Lustre: Echo OBD driver; http://www.lustre.org/ [10865.813659] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10865.845159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10881.376382] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10886.496433] LustreError: 417839:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.148@tcp: failed processing log, type 4: rc = -110 [10913.185262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10920.319924] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10928.916630] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10935.625498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.184134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10936.274968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10940.762193] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10941.414507] LustreError: 417865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10941.441207] LustreError: 417865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10946.552674] LustreError: 417864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.601728] LustreError: 417865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.631194] LustreError: 417865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10950.826700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10951.208795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10956.265153] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [10964.293941] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10970.883542] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10979.998857] Lustre: server umount lustre-OST0000 complete [10983.924253] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10983.932697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10983.947456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.978124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.982912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.101175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.118311] Lustre: Skipped 2 previous similar messages [10994.227363] Lustre: server umount lustre-MDT0000 complete [10997.218917] LustreError: 418883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10997.244068] LustreError: 418883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10998.241927] Lustre: 417179:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829078/real 1787829078] req@00000000d3e99181 x1874674530155136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787829094 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10998.299235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.319504] LustreError: 417846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829094 with bad export cookie 8053349638248144644 [10998.327595] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10998.336245] LustreError: 417846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10998.923372] Lustre: server umount lustre-MDT0001 complete [11008.488833] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [11011.280498] Key type lgssc unregistered [11011.644755] LNet: 421412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11011.656574] LNetError: 421412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11011.676302] LNet: Removed LNI 192.168.206.148@tcp [11012.674095] Key type .llcrypt unregistered [11012.677972] Key type ._llcrypt unregistered [11039.388741] Key type ._llcrypt registered [11039.390738] Key type .llcrypt registered [11039.469741] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [11040.173834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11040.240434] alg: No test for adler32 (adler32-zlib) [11041.208601] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [11041.442925] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [11043.184149] Key type lgssc registered [11044.667521] Lustre: Echo OBD driver; http://www.lustre.org/ [11053.134697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11053.165458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11054.755982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11058.760877] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11064.298411] LustreError: 422828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11064.329330] LustreError: 422828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11067.962247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.550880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11073.444601] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11081.246438] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11086.606358] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11092.946167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11093.241366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11098.831587] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11099.386026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11108.034156] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11114.465895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11114.477240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.498623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11114.985357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.009987] Lustre: Skipped 1 previous similar message [11115.050768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.068150] Lustre: Skipped 1 previous similar message [11118.572903] Lustre: server umount lustre-MDT0000 complete [11120.101143] LustreError: 422828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.110729] LustreError: 422828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11121.079396] LustreError: 422810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.48@tcp arrived at 1787829217 with bad export cookie 16865106714522506218 [11123.148507] LustreError: 422808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829219 with bad export cookie 16865106714522505322 [11123.167534] LustreError: 422808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11123.178814] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11125.221191] LustreError: 422824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.221246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11125.226190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11125.226840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11125.226844] Lustre: Skipped 1 previous similar message [11125.244058] LustreError: 422824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11128.227715] LustreError: 422823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11128.237503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11128.245322] LustreError: 422823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11129.600681] Lustre: server umount lustre-OST0000 complete [11133.353053] LustreError: 422824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.383475] LustreError: 422824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11136.725960] Lustre: server umount lustre-MDT0001 complete [11146.906814] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [11149.788732] Key type lgssc unregistered [11149.986449] LNet: 426516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11149.995492] LNetError: 426516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11151.015258] LNet: Removed LNI 192.168.206.148@tcp [11151.833705] Key type .llcrypt unregistered [11151.839116] Key type ._llcrypt unregistered [11175.198238] Key type ._llcrypt registered [11175.200536] Key type .llcrypt registered [11175.360762] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [11176.682416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11176.746089] alg: No test for adler32 (adler32-zlib) [11177.907458] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [11178.193316] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [11179.907134] Key type lgssc registered [11181.063440] Lustre: Echo OBD driver; http://www.lustre.org/ [11188.701296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11188.718583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.227390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11194.599489] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11200.481445] LustreError: 427933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11200.508041] LustreError: 427933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11203.162892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.547250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11208.233780] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11216.063868] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.024660] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11229.610536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.997681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11235.142285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11236.407877] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11244.870292] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11249.634817] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11249.643841] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.660487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11251.734047] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.48@tcp (stopping) [11253.730755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11253.749718] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.057165] Lustre: server umount lustre-MDT0001 complete [11256.856759] LustreError: 430493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11258.867373] LustreError: 427932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11258.885822] LustreError: 427932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11263.969262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.971531] LustreError: 428661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11263.990196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11264.026631] Lustre: Skipped 1 previous similar message [11268.043312] Lustre: server umount lustre-OST0000 complete [11269.095989] LustreError: 427932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11272.496111] Lustre: server umount lustre-MDT0000 complete [11285.783329] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [11288.331261] Key type lgssc unregistered [11288.659692] LNet: 431620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11288.664722] LNetError: 431620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11288.679518] LNet: Removed LNI 192.168.206.148@tcp [11289.520890] Key type .llcrypt unregistered [11289.527077] Key type ._llcrypt unregistered [11315.642472] Key type ._llcrypt registered [11315.647194] Key type .llcrypt registered [11315.802102] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [11332.586961] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [11334.223326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11334.425207] alg: No test for adler32 (adler32-zlib) [11335.546941] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [11335.889683] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [11337.786368] Key type lgssc registered [11339.131486] Lustre: Echo OBD driver; http://www.lustre.org/ [11379.328632] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [11397.363620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11397.421258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.649919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11398.690201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11398.787510] Lustre: lustre-MDT0000: new disk, initializing [11398.871992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11398.896114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11403.282809] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11419.052228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11419.376582] Lustre: lustre-OST0000: new disk, initializing [11419.380305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11419.385616] Lustre: Skipped 1 previous similar message [11419.389608] Lustre: 436658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11419.452417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11426.157404] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11429.578765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11429.592074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11429.631251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11436.552509] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11452.775580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.054573] Lustre: 435639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11453.162245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11453.265857] Lustre: lustre-MDT0001: new disk, initializing [11453.414975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11453.474532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11453.493175] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11457.982933] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11467.516284] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11467.726705] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11471.805434] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11472.003588] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11478.592749] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.48@tcp (stopping) [11480.545725] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11480.549996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11480.561355] Lustre: Skipped 1 previous similar message [11482.966694] Lustre: server umount lustre-MDT0001 complete [11490.791746] LustreError: 435646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11490.812492] LustreError: 435646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11495.910206] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.942226] LustreError: 435648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11501.025640] LustreError: 435651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11501.027648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.068253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11501.071910] Lustre: Skipped 1 previous similar message [11506.150051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11506.150220] LustreError: 435652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.266363] LustreError: 435651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.267781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11515.362087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11515.626487] Lustre: server umount lustre-OST0000 complete [11525.903328] Lustre: server umount lustre-MDT0000 complete [11540.742597] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [11543.561046] Key type lgssc unregistered [11543.922693] LNet: 440409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11543.938274] LNetError: 440409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11543.964908] LNet: Removed LNI 192.168.206.148@tcp [11545.140746] Key type .llcrypt unregistered [11545.145928] Key type ._llcrypt unregistered [11589.724345] Key type ._llcrypt registered [11589.730545] Key type .llcrypt registered [11589.860558] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [11590.173588] Key type .llcrypt unregistered [11590.175646] Key type ._llcrypt unregistered [11606.413541] Key type ._llcrypt registered [11606.417760] Key type .llcrypt registered [11606.605559] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [11608.147212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11608.184267] alg: No test for adler32 (adler32-zlib) [11609.223094] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [11609.453366] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [11611.120208] Key type lgssc registered [11612.213965] Lustre: Echo OBD driver; http://www.lustre.org/ [11638.245075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11638.284442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11639.495316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11639.536768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11639.669787] Lustre: lustre-MDT0000: new disk, initializing [11639.812627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11639.852715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11644.028508] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11653.885646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11654.145157] Lustre: 443795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11654.306910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11654.313414] Lustre: Skipped 1 previous similar message [11654.409518] Lustre: lustre-MDT0001: new disk, initializing [11654.500807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11654.555108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11654.565144] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11660.277363] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11670.776886] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11677.239830] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11683.586818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11683.791796] Lustre: lustre-OST0000: new disk, initializing [11683.794935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11683.803329] Lustre: 445765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11683.922152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11689.585698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11689.599152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11689.716163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11690.053626] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11699.231836] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11705.313498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11705.317981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11705.327146] Lustre: Skipped 2 previous similar messages [11705.329544] Lustre: Skipped 1 previous similar message [11710.436135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11710.441855] Lustre: Skipped 3 previous similar messages [11713.403655] Lustre: server umount lustre-MDT0000 complete [11717.224430] LustreError: 443786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829813 with bad export cookie 16978471946438886502 [11717.224766] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11717.243121] LustreError: 443786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11717.720111] Lustre: server umount lustre-MDT0001 complete [11731.679642] Lustre: server umount lustre-OST0000 complete [11743.579240] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:23:58 (1787829838) [11751.764648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11752.244638] LustreError: 447793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.258054] LustreError: 447793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11752.299885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11756.164276] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11757.540932] LustreError: 447794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.661554] LustreError: 447793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.668555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11765.146891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11769.025580] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11776.890762] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11782.529118] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11802.152843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11802.434687] Lustre: Found index 0 for lustre-OST0000, updating log [11802.627928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.913415] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [11816.020440] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11823.144254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11831.265475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11831.279123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.297377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11837.401447] Lustre: server umount lustre-OST0000 complete [11853.793208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.795344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11853.807678] Lustre: Skipped 2 previous similar messages [11853.822376] Lustre: Skipped 4 previous similar messages [11857.890458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11857.899265] Lustre: Skipped 1 previous similar message [11859.478460] Lustre: server umount lustre-MDT0000 complete [11863.496189] LustreError: 447774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829959 with bad export cookie 16978471946438887895 [11863.515038] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.520226] LustreError: 447774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11863.864203] Lustre: server umount lustre-MDT0001 complete [11871.994532] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:26:07 (1787829967) [11872.674480] LNet: 1 local NIs in recovery (showing 1): 192.168.206.148@tcp [12490.296424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.780187] LustreError: 454226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12490.908882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12495.635488] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [12496.359303] LustreError: 454227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12500.451549] LustreError: 454226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.694650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12504.095595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12508.162140] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [12515.267340] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12520.655377] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12540.016859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12540.210139] Lustre: Found index 0 for lustre-OST0000, updating log [12540.452764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12546.417962] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [12550.560821] LNet: 1 local NIs in recovery (showing 1): 192.168.206.148@tcp [12553.442851] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12555.746340] Lustre: server umount lustre-OST0000 complete [12560.295070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.299134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12560.301553] Lustre: Skipped 1 previous similar message [12560.321256] Lustre: Skipped 3 previous similar messages [12561.314230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12561.319420] Lustre: Skipped 1 previous similar message [12563.365543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12563.380331] Lustre: Skipped 3 previous similar messages [12565.707790] Lustre: server umount lustre-MDT0000 complete [12566.435435] LustreError: 454223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12566.472639] LustreError: 454223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12569.530234] LustreError: 454961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830665 with bad export cookie 16978471946438888973 [12569.543144] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12569.544886] LustreError: 454961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12569.937708] Lustre: server umount lustre-MDT0001 complete [12600.165270] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [12610.095236] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [12648.781676] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:39:03 (1787830743) [12669.697944] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [12672.526375] Key type lgssc unregistered [12672.860404] LNet: 460838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12672.864499] LNetError: 460838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12801.900330] LNet: Removed LNI 192.168.206.148@tcp [12803.055702] Key type .llcrypt unregistered [12803.057870] Key type ._llcrypt unregistered [12817.917413] Key type ._llcrypt registered [12817.924950] Key type .llcrypt registered [12818.026659] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [12818.785646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12818.924251] alg: No test for adler32 (adler32-zlib) [12819.978551] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [12820.241581] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [12821.944276] Key type lgssc registered [12823.097295] Lustre: Echo OBD driver; http://www.lustre.org/ [12827.550718] LNet: Added LNI 192.168.206.148@tcp100 [8/256/0/180] [12828.391752] LNet: Added LNI 192.168.206.148@tcp101 [8/256/0/180] [12829.113900] LNet: Added LNI 192.168.206.148@tcp102 [8/256/0/180] [12829.896726] LNet: Added LNI 192.168.206.148@tcp103 [8/256/0/180] [12830.757954] LNet: Added LNI 192.168.206.148@tcp104 [8/256/0/180] [12831.619103] LNet: Added LNI 192.168.206.148@tcp105 [8/256/0/180] [12832.390219] LNet: Added LNI 192.168.206.148@tcp106 [8/256/0/180] [12833.140298] LNet: Added LNI 192.168.206.148@tcp107 [8/256/0/180] [12833.922619] LNet: Added LNI 192.168.206.148@tcp108 [8/256/0/180] [12834.830684] LNet: Added LNI 192.168.206.148@tcp109 [8/256/0/180] [12835.685882] LNet: Added LNI 192.168.206.148@tcp110 [8/256/0/180] [12836.620313] LNet: Added LNI 192.168.206.148@tcp111 [8/256/0/180] [12837.491267] LNet: Added LNI 192.168.206.148@tcp112 [8/256/0/180] [12838.227817] LNet: Added LNI 192.168.206.148@tcp113 [8/256/0/180] [12839.081047] LNet: Added LNI 192.168.206.148@tcp114 [8/256/0/180] [12839.929263] LNet: Added LNI 192.168.206.148@tcp115 [8/256/0/180] [12840.687477] LNet: Added LNI 192.168.206.148@tcp116 [8/256/0/180] [12841.424582] LNet: Added LNI 192.168.206.148@tcp117 [8/256/0/180] [12842.248600] LNet: Added LNI 192.168.206.148@tcp118 [8/256/0/180] [12842.892669] LNet: Added LNI 192.168.206.148@tcp119 [8/256/0/180] [12843.694592] LNet: Added LNI 192.168.206.148@tcp120 [8/256/0/180] [12844.528122] LNet: Added LNI 192.168.206.148@tcp121 [8/256/0/180] [12845.203596] LNet: Added LNI 192.168.206.148@tcp122 [8/256/0/180] [12845.944121] LNet: Added LNI 192.168.206.148@tcp123 [8/256/0/180] [12846.764174] LNet: Added LNI 192.168.206.148@tcp124 [8/256/0/180] [12847.713366] LNet: Added LNI 192.168.206.148@tcp125 [8/256/0/180] [12848.594117] LNet: Added LNI 192.168.206.148@tcp126 [8/256/0/180] [12849.497344] LNet: Added LNI 192.168.206.148@tcp127 [8/256/0/180] [12850.309890] LNet: Added LNI 192.168.206.148@tcp128 [8/256/0/180] [12851.102028] LNet: Added LNI 192.168.206.148@tcp129 [8/256/0/180] [12851.859041] LNet: Added LNI 192.168.206.148@tcp130 [8/256/0/180] [12852.659654] LNet: Added LNI 192.168.206.148@tcp131 [8/256/0/180] [12853.539558] LNet: Added LNI 192.168.206.148@tcp132 [8/256/0/180] [12854.424702] LNet: Added LNI 192.168.206.148@tcp133 [8/256/0/180] [12855.165066] LNet: Added LNI 192.168.206.148@tcp134 [8/256/0/180] [12855.937492] LNet: Added LNI 192.168.206.148@tcp135 [8/256/0/180] [12861.975820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12862.008103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12863.146651] Lustre: 463872:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.148@tcp: can't add NID 192.168.206.148@tcp130: rc = -75 [12863.240877] LNetError: 461330:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12863.253991] LNetError: Protocol error connecting to 192.168.206.148@tcp117 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12863.269185] LNetError: Skipped 6 previous similar messages [12863.272896] LNet: 5 local NIs in recovery (showing 5): 192.168.206.148@tcp121, 192.168.206.148@tcp122, 192.168.206.148@tcp124, 192.168.206.148@tcp125, 192.168.206.148@tcp126 [12863.329463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12863.359527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12863.476702] Lustre: lustre-MDT0000: new disk, initializing [12863.546175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12863.563928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12864.296249] LNetError: 461333:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12864.304785] LNetError: 461333:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [12864.315457] LNetError: Protocol error connecting to 192.168.206.148@tcp109 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12864.327697] LNetError: Skipped 7 previous similar messages [12865.377282] LNetError: 463903:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12865.392423] LNetError: 463903:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [12865.407425] LNetError: Protocol error connecting to 192.168.206.148@tcp125 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12868.039931] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [12870.435825] LNetError: 461331:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12870.454101] LNetError: 461331:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [12870.464713] LNetError: Protocol error connecting to 192.168.206.148@tcp128 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12870.478600] LNetError: Skipped 3 previous similar messages [12877.863211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12878.026814] Lustre: 463888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12878.104303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12878.108595] Lustre: Skipped 1 previous similar message [12878.208884] Lustre: lustre-MDT0001: new disk, initializing [12878.262869] Lustre: 464690:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.148@tcp: can't add NID 192.168.206.148@tcp130: rc = -75 [12878.268229] Lustre: 464690:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12878.357647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12878.390377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12878.405672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12878.628884] LNetError: 461330:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12878.641608] LNetError: Protocol error connecting to 192.168.206.148@tcp128 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12879.920035] Lustre: 464775:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.148@tcp: can't add NID 192.168.206.148@tcp130: rc = -75 [12879.928591] Lustre: 464775:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12883.060606] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [12893.562631] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12895.010830] LNetError: 463967:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12895.031260] LNetError: Protocol error connecting to 192.168.206.148@tcp128 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12899.829195] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12906.077041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12906.298672] Lustre: lustre-OST0000: new disk, initializing [12906.303485] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12906.310598] Lustre: 465863:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12906.411683] Lustre: 465868:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.148@tcp: can't add NID 192.168.206.148@tcp130: rc = -75 [12906.418243] Lustre: 465868:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12906.488257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12911.394480] LNetError: 461330:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12911.415356] LNetError: 461330:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [12911.432693] LNetError: Protocol error connecting to 192.168.206.148@tcp129 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12911.443424] LNetError: Skipped 6 previous similar messages [12912.068456] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [12912.185470] Lustre: 466219:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.148@tcp: can't add NID 192.168.206.148@tcp130: rc = -75 [12912.197844] Lustre: 466219:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12912.278148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12912.290662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12912.496247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12922.646559] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12923.680116] LNet: 1 local NIs in recovery (showing 1): 192.168.206.148@tcp128 [12933.089660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12933.101498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.107695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12933.121540] Lustre: Skipped 1 previous similar message [12933.138463] Lustre: Skipped 1 previous similar message [12938.215321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12938.232954] Lustre: Skipped 1 previous similar message [12943.332567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12943.338618] Lustre: Skipped 1 previous similar message [12944.352161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12944.569398] Lustre: server umount lustre-OST0000 complete [12948.451151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12948.467662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12948.475196] Lustre: Skipped 1 previous similar message [12953.391630] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.148 [12953.403425] LNetError: 463892:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [12953.412652] LNetError: Protocol error connecting to 192.168.206.148@tcp109 at host 192.168.206.148:988: is it running a compatible version of Lustre? [12953.436271] LNetError: Skipped 16 previous similar messages [12953.570400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12953.577967] Lustre: Skipped 1 previous similar message [12954.695839] Lustre: server umount lustre-MDT0000 complete [12958.520845] LustreError: 463881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831054 with bad export cookie 9872064629360456368 [12958.528550] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12958.542338] LustreError: 463881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12958.956985] Lustre: server umount lustre-MDT0001 complete [12972.373391] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [12975.389830] Key type lgssc unregistered [12975.806337] LNet: 467781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12975.815095] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12975.825183] LNet: Removed LNI 192.168.206.148@tcp [12976.867198] LNet: Removed LNI 192.168.206.148@tcp100 [12976.869510] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12976.880994] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12976.904682] LNet: Removed LNI 192.168.206.148@tcp101 [12977.956671] LNet: Removed LNI 192.168.206.148@tcp102 [12977.971397] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12977.980462] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12977.992369] LNet: Removed LNI 192.168.206.148@tcp103 [12979.040227] LNet: Removed LNI 192.168.206.148@tcp104 [12980.064173] LNet: Removed LNI 192.168.206.148@tcp105 [12980.065970] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12980.083257] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12980.103806] LNet: Removed LNI 192.168.206.148@tcp106 [12981.152650] LNet: Removed LNI 192.168.206.148@tcp107 [12982.176139] LNet: Removed LNI 192.168.206.148@tcp108 [12983.200767] LNet: Removed LNI 192.168.206.148@tcp109 [12984.224146] LNet: Removed LNI 192.168.206.148@tcp110 [12984.231231] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12984.239658] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12984.244688] LNet: Removed LNI 192.168.206.148@tcp111 [12985.254434] LNet: Removed LNI 192.168.206.148@tcp112 [12986.272138] LNet: Removed LNI 192.168.206.148@tcp113 [12987.296415] LNet: Removed LNI 192.168.206.148@tcp114 [12988.320552] LNet: Removed LNI 192.168.206.148@tcp115 [12989.344233] LNet: Removed LNI 192.168.206.148@tcp116 [12990.368150] LNet: Removed LNI 192.168.206.148@tcp117 [12991.392687] LNet: Removed LNI 192.168.206.148@tcp118 [12992.416164] LNet: Removed LNI 192.168.206.148@tcp119 [12992.424770] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12992.440185] LNetError: 467781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12992.449205] LNet: Removed LNI 192.168.206.148@tcp120 [12993.504140] LNet: Removed LNI 192.168.206.148@tcp121 [12994.528238] LNet: Removed LNI 192.168.206.148@tcp122 [12995.552571] LNet: Removed LNI 192.168.206.148@tcp123 [12996.576145] LNet: Removed LNI 192.168.206.148@tcp124 [12997.600832] LNet: Removed LNI 192.168.206.148@tcp125 [12998.624348] LNet: Removed LNI 192.168.206.148@tcp126 [12999.648161] LNet: Removed LNI 192.168.206.148@tcp127 [12999.650993] LNet: Removed LNI 192.168.206.148@tcp128 [13000.672151] LNet: Removed LNI 192.168.206.148@tcp129 [13000.677800] LNet: Removed LNI 192.168.206.148@tcp130 [13000.684643] LNet: Removed LNI 192.168.206.148@tcp131 [13000.689925] LNet: Removed LNI 192.168.206.148@tcp132 [13000.697523] LNet: Removed LNI 192.168.206.148@tcp133 [13000.704457] LNet: Removed LNI 192.168.206.148@tcp134 [13000.722762] LNet: Removed LNI 192.168.206.148@tcp135 [13001.481883] Key type .llcrypt unregistered [13001.485424] Key type ._llcrypt unregistered [13027.223980] Key type ._llcrypt registered [13027.226799] Key type .llcrypt registered [13027.332042] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [13042.460792] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [13043.894681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13043.986691] alg: No test for adler32 (adler32-zlib) [13045.319236] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [13045.642309] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [13047.392140] Key type lgssc registered [13048.340465] Lustre: Echo OBD driver; http://www.lustre.org/ [13081.891367] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:46:16 (1787831176) [13098.023928] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [13100.362395] Key type lgssc unregistered [13100.625628] LNet: 471873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13100.636385] LNetError: 471873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13100.651715] LNet: Removed LNI 192.168.206.148@tcp [13101.375417] Key type .llcrypt unregistered [13101.377358] Key type ._llcrypt unregistered [13117.036969] Key type ._llcrypt registered [13117.039442] Key type .llcrypt registered [13117.153116] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [13118.006968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13118.189903] alg: No test for adler32 (adler32-zlib) [13119.272817] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [13119.484259] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [13121.123400] Key type lgssc registered [13122.225818] Lustre: Echo OBD driver; http://www.lustre.org/ [13130.377317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13130.447423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13131.700873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13131.732879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13131.812297] Lustre: lustre-MDT0000: new disk, initializing [13131.885309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13131.905463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13135.468916] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13144.462376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13144.599883] Lustre: 473088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13144.680585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13144.685981] Lustre: Skipped 1 previous similar message [13144.765656] Lustre: lustre-MDT0001: new disk, initializing [13144.850802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13144.878069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13144.888541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13148.055862] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13156.722345] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13163.301970] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13171.175192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13171.548506] Lustre: lustre-OST0000: new disk, initializing [13171.553805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13171.562231] Lustre: 475056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13171.651525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13177.527144] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13177.907672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13177.917954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13178.021301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13185.769805] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13189.077820] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing lnet_if_list [13194.294749] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13194.321759] LNet: Added LNI 192.168.206.148@tcp10 [8/256/0/180] [13199.807682] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13199.858717] LNet: Added LNI 192.168.206.148@tcp11 [8/256/0/180] [13203.497286] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13203.520199] LNet: Added LNI 192.168.206.148@tcp12 [8/256/0/180] [13207.634951] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13207.668374] LNet: Added LNI 192.168.206.148@tcp13 [8/256/0/180] [13211.612084] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13211.642199] LNet: Added LNI 192.168.206.148@tcp14 [8/256/0/180] [13216.036850] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13216.072718] LNet: Added LNI 192.168.206.148@tcp15 [8/256/0/180] [13219.502757] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13219.519895] LNet: Added LNI 192.168.206.148@tcp16 [8/256/0/180] [13223.542793] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13223.558932] LNet: Added LNI 192.168.206.148@tcp17 [8/256/0/180] [13227.724366] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13227.762249] LNet: Added LNI 192.168.206.148@tcp18 [8/256/0/180] [13232.149653] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13232.182903] LNet: Added LNI 192.168.206.148@tcp19 [8/256/0/180] [13236.764894] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13236.800678] LNet: Added LNI 192.168.206.148@tcp20 [8/256/0/180] [13242.014763] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13242.057154] LNet: Added LNI 192.168.206.148@tcp21 [8/256/0/180] [13247.859820] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13247.894806] LNet: Added LNI 192.168.206.148@tcp22 [8/256/0/180] [13254.001610] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13254.043681] LNet: Added LNI 192.168.206.148@tcp23 [8/256/0/180] [13259.748616] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13259.791681] LNet: Added LNI 192.168.206.148@tcp24 [8/256/0/180] [13265.742505] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13265.774348] LNet: Added LNI 192.168.206.148@tcp25 [8/256/0/180] [13270.334767] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13270.363866] LNet: Added LNI 192.168.206.148@tcp26 [8/256/0/180] [13275.427567] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13275.457143] LNet: Added LNI 192.168.206.148@tcp27 [8/256/0/180] [13282.072763] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13282.110209] LNet: Added LNI 192.168.206.148@tcp28 [8/256/0/180] [13288.216058] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13288.247447] LNet: Added LNI 192.168.206.148@tcp29 [8/256/0/180] [13294.753748] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13294.788655] LNet: Added LNI 192.168.206.148@tcp30 [8/256/0/180] [13300.714154] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13300.755640] LNet: Added LNI 192.168.206.148@tcp31 [8/256/0/180] [13306.878306] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13306.906288] LNet: Added LNI 192.168.206.148@tcp32 [8/256/0/180] [13313.049107] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13313.077040] LNet: Added LNI 192.168.206.148@tcp33 [8/256/0/180] [13319.720296] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13319.754035] LNet: Added LNI 192.168.206.148@tcp34 [8/256/0/180] [13326.533829] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13326.563107] LNet: Added LNI 192.168.206.148@tcp35 [8/256/0/180] [13333.119902] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13333.153602] LNet: Added LNI 192.168.206.148@tcp36 [8/256/0/180] [13338.778085] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13338.799907] LNet: Added LNI 192.168.206.148@tcp37 [8/256/0/180] [13344.780132] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13344.808556] LNet: Added LNI 192.168.206.148@tcp38 [8/256/0/180] [13350.827188] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13350.848940] LNet: Added LNI 192.168.206.148@tcp39 [8/256/0/180] [13356.587275] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13356.609380] LNet: Added LNI 192.168.206.148@tcp40 [8/256/0/180] [13361.949672] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13361.976433] LNet: Added LNI 192.168.206.148@tcp41 [8/256/0/180] [13368.537459] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13368.576449] LNet: Added LNI 192.168.206.148@tcp42 [8/256/0/180] [13375.099703] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13375.125742] LNet: Added LNI 192.168.206.148@tcp43 [8/256/0/180] [13381.104908] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13381.120856] LNet: Added LNI 192.168.206.148@tcp44 [8/256/0/180] [13388.563240] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13388.600820] LNet: Added LNI 192.168.206.148@tcp45 [8/256/0/180] [13396.653511] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13396.713671] LNet: Added LNI 192.168.206.148@tcp46 [8/256/0/180] [13404.425652] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13404.478500] LNet: Added LNI 192.168.206.148@tcp47 [8/256/0/180] [13412.199307] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13412.248876] LNet: Added LNI 192.168.206.148@tcp48 [8/256/0/180] [13420.558460] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13420.594757] LNet: Added LNI 192.168.206.148@tcp49 [8/256/0/180] [13428.975885] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13429.016052] LNet: Added LNI 192.168.206.148@tcp50 [8/256/0/180] [13445.156674] Lustre: server umount lustre-MDT0000 complete [13447.141517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13447.147650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13448.565137] LustreError: 473078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831544 with bad export cookie 17743438465596443426 [13448.568256] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13448.576654] LustreError: 473078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13448.854603] Lustre: server umount lustre-MDT0001 complete [13465.059651] Lustre: 472357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831545/real 1787831545] req@00000000c98c660e x1874676904259456/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13465.104875] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.826063] Lustre: 472359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831545/real 1787831545] req@00000000b30401ff x1874676904259584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831561 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13466.459831] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [13469.153196] Lustre: 472360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831549/real 1787831549] req@000000005488dc6c x1874676904259712/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13474.274764] Lustre: 472357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831554/real 1787831554] req@0000000096e9e81d x1874676904260096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13474.311196] Lustre: 472357:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13475.944381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13479.392653] Lustre: 472358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787831559/real 1787831559] req@00000000ba2bd826 x1874676904260480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787831575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13479.426209] Lustre: 472358:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13493.537450] LustreError: 472356:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000745b4b45 x1874676904261632/t0(0) o250->MGC192.168.206.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13493.803308] LustreError: 475048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13493.821409] LustreError: 475048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13493.867691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13495.909743] LustreError: 475049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13496.936246] LustreError: 475048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13498.716365] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13502.434461] LustreError: 475049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13507.451080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13507.558211] LustreError: 475048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13507.829539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13509.868658] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13510.891969] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13512.797546] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13519.704547] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13524.849109] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13543.931656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13543.949961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13543.979069] Lustre: Skipped 1 previous similar message [13543.988082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13544.940300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13544.948890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13546.980216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13546.986489] Lustre: Skipped 1 previous similar message [13552.101404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13552.110133] Lustre: Skipped 1 previous similar message [13557.225313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13557.232204] Lustre: Skipped 1 previous similar message [13558.240197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13558.376409] Lustre: server umount lustre-OST0000 complete [13562.340551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13562.351054] Lustre: Skipped 2 previous similar messages [13567.337379] Lustre: server umount lustre-MDT0000 complete [13567.463347] LustreError: 499295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13567.484777] LustreError: 499295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13570.554112] LustreError: 473080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831666 with bad export cookie 17743438465596446058 [13570.555909] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13570.569126] LustreError: 473080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13570.849364] Lustre: server umount lustre-MDT0001 complete [13580.370901] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [13582.898088] Key type lgssc unregistered [13583.363270] LNet: 502093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13583.370517] LNetError: 502093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13584.416161] LNet: Removed LNI 192.168.206.148@tcp [13584.421512] LNetError: 502093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13584.434653] LNet: Removed LNI 192.168.206.148@tcp10 [13584.443174] LNet: Removed LNI 192.168.206.148@tcp11 [13584.450106] LNet: Removed LNI 192.168.206.148@tcp12 [13584.459587] LNet: Removed LNI 192.168.206.148@tcp13 [13584.461477] LNet: Removed LNI 192.168.206.148@tcp14 [13584.470506] LNet: Removed LNI 192.168.206.148@tcp15 [13584.475859] LNet: Removed LNI 192.168.206.148@tcp16 [13584.478687] LNet: Removed LNI 192.168.206.148@tcp17 [13584.482766] LNet: Removed LNI 192.168.206.148@tcp18 [13584.490924] LNet: Removed LNI 192.168.206.148@tcp19 [13584.494494] LNet: Removed LNI 192.168.206.148@tcp20 [13584.497535] LNet: Removed LNI 192.168.206.148@tcp21 [13584.502061] LNet: Removed LNI 192.168.206.148@tcp22 [13584.508744] LNet: Removed LNI 192.168.206.148@tcp23 [13584.512241] LNet: Removed LNI 192.168.206.148@tcp24 [13584.520714] LNet: Removed LNI 192.168.206.148@tcp25 [13584.522592] LNet: Removed LNI 192.168.206.148@tcp26 [13584.530185] LNet: Removed LNI 192.168.206.148@tcp27 [13584.535412] LNet: Removed LNI 192.168.206.148@tcp28 [13584.542101] LNet: Removed LNI 192.168.206.148@tcp29 [13584.545868] LNet: Removed LNI 192.168.206.148@tcp30 [13584.548917] LNet: Removed LNI 192.168.206.148@tcp31 [13584.551552] LNet: Removed LNI 192.168.206.148@tcp32 [13584.554506] LNet: Removed LNI 192.168.206.148@tcp33 [13584.557021] LNet: Removed LNI 192.168.206.148@tcp34 [13584.559220] LNet: Removed LNI 192.168.206.148@tcp35 [13584.562210] LNet: Removed LNI 192.168.206.148@tcp36 [13584.565347] LNet: Removed LNI 192.168.206.148@tcp37 [13584.569491] LNet: Removed LNI 192.168.206.148@tcp38 [13584.573130] LNet: Removed LNI 192.168.206.148@tcp39 [13584.576749] LNet: Removed LNI 192.168.206.148@tcp40 [13584.582138] LNet: Removed LNI 192.168.206.148@tcp41 [13584.584180] LNet: Removed LNI 192.168.206.148@tcp42 [13584.590775] LNet: Removed LNI 192.168.206.148@tcp43 [13584.592454] LNet: Removed LNI 192.168.206.148@tcp44 [13584.603390] LNet: Removed LNI 192.168.206.148@tcp45 [13584.610675] LNet: Removed LNI 192.168.206.148@tcp46 [13584.614616] LNet: Removed LNI 192.168.206.148@tcp47 [13584.622454] LNet: Removed LNI 192.168.206.148@tcp48 [13584.626749] LNet: Removed LNI 192.168.206.148@tcp49 [13584.635925] LNet: Removed LNI 192.168.206.148@tcp50 [13585.325390] Key type .llcrypt unregistered [13585.327089] Key type ._llcrypt unregistered [13602.850432] Key type ._llcrypt registered [13602.852926] Key type .llcrypt registered [13602.965102] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [13614.451642] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [13615.186834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13615.212188] alg: No test for adler32 (adler32-zlib) [13616.203710] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [13616.406474] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [13618.048180] Key type lgssc registered [13619.029514] Lustre: Echo OBD driver; http://www.lustre.org/ [13649.226515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13649.250320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13650.511782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13650.534792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13650.599203] Lustre: lustre-MDT0000: new disk, initializing [13650.683407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13650.699642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13654.826736] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13664.399089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13664.551288] Lustre: 505275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13664.599673] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13664.603775] Lustre: Skipped 1 previous similar message [13664.662289] Lustre: lustre-MDT0001: new disk, initializing [13664.772468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13664.817975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13664.827772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13669.083946] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13677.258341] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13682.644250] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13690.035947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13690.251850] Lustre: *** cfs_fail_loc=90f, val=0*** [13690.331658] Lustre: lustre-OST0000: new disk, initializing [13690.337580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13690.347519] Lustre: 507289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13690.443296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13696.313935] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13699.177276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13699.192831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13699.262235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13704.569023] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13714.916146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13714.924752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13714.932292] LustreError: Skipped 1 previous similar message [13714.958354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13714.968938] Lustre: Skipped 1 previous similar message [13716.584327] Lustre: server umount lustre-OST0000 complete [13720.035231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13720.036847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13720.046346] Lustre: Skipped 2 previous similar messages [13725.153589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13725.157935] Lustre: Skipped 2 previous similar messages [13730.275544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13730.280638] Lustre: Skipped 1 previous similar message [13734.368246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13734.571350] Lustre: server umount lustre-MDT0000 complete [13737.861752] LustreError: 506068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831834 with bad export cookie 12266427346606885892 [13737.868372] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13737.878034] LustreError: 506068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13738.206401] Lustre: server umount lustre-MDT0001 complete [13748.692502] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [13751.586872] Key type lgssc unregistered [13751.908105] LNet: 509199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13751.926449] LNetError: 509199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13751.945753] LNet: Removed LNI 192.168.206.148@tcp [13752.514226] Key type .llcrypt unregistered [13752.516667] Key type ._llcrypt unregistered [13772.568440] Key type ._llcrypt registered [13772.569958] Key type .llcrypt registered [13772.664803] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [13773.450730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13773.504116] alg: No test for adler32 (adler32-zlib) [13774.556755] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [13774.787327] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [13776.497099] Key type lgssc registered [13777.611852] Lustre: Echo OBD driver; http://www.lustre.org/ [13784.694946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13784.718467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13786.140236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13790.456926] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13796.328913] LustreError: 510469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13796.356427] LustreError: 510469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13798.404897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13798.703312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13802.301855] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13808.477522] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13815.274826] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13822.387624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13822.784533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13828.217388] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13835.515698] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13844.209728] loop0: detected capacity change from 0 to 204800000 [13844.413893] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13844.423956] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13844.434740] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13849.541743] loop0: detected capacity change from 0 to 204800000 [13849.677636] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13849.692762] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13849.702178] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13853.960660] loop0: detected capacity change from 0 to 204800000 [13858.580192] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13858.692536] Lustre: fs15246-OST0000: new disk, initializing [13858.697097] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13858.702153] Lustre: 514062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13858.744401] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13862.693988] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13871.346813] loop1: detected capacity change from 0 to 204800000 [13876.032187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13876.091425] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13876.169316] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13876.270278] Lustre: fs15246-MDT0000: new disk, initializing [13876.476462] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13876.527903] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13876.733973] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13877.796900] Lustre: 515185:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008c56b14d x1874677590961408/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787831989 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13881.076473] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [13882.863946] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13882.869049] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13882.917629] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13902.830077] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13902.846038] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13904.987282] Lustre: server umount fs15246-MDT0000 complete [13907.956109] LustreError: 510464:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13913.068866] LustreError: 515934:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13914.039373] Lustre: server umount fs15246-OST0000 complete [13926.371410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13926.380305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13926.388490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13928.418273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13928.425413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13928.442737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13930.845419] Lustre: server umount lustre-OST0000 complete [13937.121699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13937.135371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.152970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13937.158281] Lustre: Skipped 1 previous similar message [13940.708787] Lustre: server umount lustre-MDT0000 complete [13943.779645] LustreError: 511198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13943.795527] LustreError: 511198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13943.813683] LustreError: 515179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832039 with bad export cookie 10751368546666431797 [13943.816275] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13943.822567] LustreError: 515179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13944.112910] Lustre: server umount lustre-MDT0001 complete [13954.993267] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [13957.335449] Key type lgssc unregistered [13957.615761] LNet: 518283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13957.632741] LNetError: 518283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13957.648084] LNet: Removed LNI 192.168.206.148@tcp [13958.367950] Key type .llcrypt unregistered [13958.371680] Key type ._llcrypt unregistered [13974.016875] Key type ._llcrypt registered [13974.022050] Key type .llcrypt registered [13974.157886] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [13975.100425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13975.212649] alg: No test for adler32 (adler32-zlib) [13976.255745] Lustre: Lustre: Build Version: 2.17.57_80_gdb9f866 [13976.477298] LNet: Added LNI 192.168.206.148@tcp [8/256/0/180] [13978.136116] Key type lgssc registered [13979.057885] Lustre: Echo OBD driver; http://www.lustre.org/ [14011.135580] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [14020.553874] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [14051.253788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14051.275199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14052.501912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14052.529477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14052.584440] Lustre: lustre-MDT0000: new disk, initializing [14052.637281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14052.653374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14056.963565] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [14067.483898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14067.627728] Lustre: 522759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14067.714976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14067.722706] Lustre: Skipped 1 previous similar message [14067.792653] Lustre: lustre-MDT0001: new disk, initializing [14067.938138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14067.994348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14068.001807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14072.326285] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [14080.867046] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14085.580630] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14091.107341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14091.302392] Lustre: lustre-OST0000: new disk, initializing [14091.305700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14091.310645] Lustre: 524726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14091.391497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14096.813637] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_default_debug -1 all [14101.086871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14101.095809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14101.193316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14105.336210] Lustre: DEBUG MARKER: oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14109.617989] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14109.841507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14113.938593] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14114.231215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14116.841916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14116.843158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14116.847063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14116.857809] Lustre: Skipped 1 previous similar message [14121.990303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14121.992773] Lustre: Skipped 1 previous similar message [14122.193474] Lustre: server umount lustre-OST0000 complete [14127.081655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14127.101746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14127.108583] Lustre: Skipped 1 previous similar message [14131.822781] Lustre: server umount lustre-MDT0000 complete [14132.194382] LustreError: 522772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14132.210715] LustreError: 522772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14134.741535] LustreError: 522752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832230 with bad export cookie 4609299072341005794 [14134.748379] LustreError: MGC192.168.206.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14134.748608] LustreError: 522752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14135.066677] Lustre: server umount lustre-MDT0001 complete [14155.297320] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing set_hostid [14165.330311] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing load_modules_local [14192.550301] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13955 sec ========= 08:04:47 (1787832287) [14193.794420] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:04:49 (1787832289) === [14196.418848] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:04:51 (1787832291) === [14234.068934] Lustre: DEBUG MARKER: oleg648-server.virtnet: executing unload_modules_local [14236.612835] Key type lgssc unregistered [14236.913043] LNet: 531369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14236.921571] LNetError: 531369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14236.954829] LNet: Removed LNI 192.168.206.148@tcp [14237.906333] Key type .llcrypt unregistered [14237.909929] Key type ._llcrypt unregistered