[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514931311 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003392] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010147] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.021083] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026775] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028056] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030130] *** VALIDATE tmpfs *** [ 0.032475] *** VALIDATE proc *** [ 0.033263] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038022] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042375] debug: unmapping init [mem 0xffffffffbc259000-0xffffffffbc260fff] [ 0.044146] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045536] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.046972] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.047952] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049010] ... fixed-purpose events: 3 [ 0.049945] ... event mask: 000000070000000f [ 0.050326] rcu: Hierarchical SRCU implementation. [ 0.052222] smp: Bringing up secondary CPUs ... [ 0.053461] x86: Booting SMP configuration: [ 0.054029] .... node #0, CPUs: #1 #2 #3 [ 0.057605] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060025] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.249273] node 0 deferred pages initialised in 185ms [ 0.253089] devtmpfs: initialized [ 0.254366] x86/mm: Memory block size: 128MB [ 0.257998] gcov: version magic: 0x41383552 [ 0.260406] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.261083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.262418] pinctrl core: initialized pinctrl subsystem [ 0.263000] [ 0.265014] ************************************************************* [ 0.267012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268011] ** ** [ 0.270013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.271009] ** ** [ 0.273020] ** This means that this kernel is built to expose internal ** [ 0.275016] ** IOMMU data structures, which may compromise security on ** [ 0.276028] ** your system. ** [ 0.277011] ** ** [ 0.279018] ** If you see this message and you are not debugging the ** [ 0.281017] ** kernel, report this immediately to your vendor! ** [ 0.283017] ** ** [ 0.285014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.287011] ************************************************************* [ 0.289780] NET: Registered protocol family 16 [ 0.291498] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297145] cpuidle: using governor menu [ 0.298711] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.300649] PCI: Using configuration type 1 for base access [ 0.302175] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.313036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315061] cryptd: max_cpu_qlen set to 1000 [ 0.318353] ACPI: Added _OSI(Module Device) [ 0.320023] ACPI: Added _OSI(Processor Device) [ 0.321027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.323023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329013] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335422] ACPI: Interpreter enabled [ 0.337085] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338015] ACPI: Using IOAPIC for interrupt routing [ 0.340147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.344468] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.354000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.357059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.359028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.363110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.369683] acpiphp: Slot [2] registered [ 0.371146] acpiphp: Slot [5] registered [ 0.372135] acpiphp: Slot [6] registered [ 0.374177] acpiphp: Slot [7] registered [ 0.375180] acpiphp: Slot [8] registered [ 0.377146] acpiphp: Slot [9] registered [ 0.378144] acpiphp: Slot [10] registered [ 0.380156] acpiphp: Slot [3] registered [ 0.381114] acpiphp: Slot [4] registered [ 0.383187] acpiphp: Slot [11] registered [ 0.384105] acpiphp: Slot [12] registered [ 0.386126] acpiphp: Slot [13] registered [ 0.387125] acpiphp: Slot [14] registered [ 0.389137] acpiphp: Slot [15] registered [ 0.390126] acpiphp: Slot [16] registered [ 0.392175] acpiphp: Slot [17] registered [ 0.393120] acpiphp: Slot [18] registered [ 0.395188] acpiphp: Slot [19] registered [ 0.396177] acpiphp: Slot [20] registered [ 0.398139] acpiphp: Slot [21] registered [ 0.400113] acpiphp: Slot [22] registered [ 0.401149] acpiphp: Slot [23] registered [ 0.403126] acpiphp: Slot [24] registered [ 0.405263] acpiphp: Slot [25] registered [ 0.407129] acpiphp: Slot [26] registered [ 0.408148] acpiphp: Slot [27] registered [ 0.410216] acpiphp: Slot [28] registered [ 0.412148] acpiphp: Slot [29] registered [ 0.414134] acpiphp: Slot [30] registered [ 0.415169] acpiphp: Slot [31] registered [ 0.417133] PCI host bridge to bus 0000:00 [ 0.418029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.426050] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.429045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.431042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.433219] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.436800] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440473] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.449023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.455044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.457042] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.459024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.462022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.465664] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.469833] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.472055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.476013] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.482026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.495020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.501023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508418] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.526026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.537032] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.552024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.566654] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.576023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.586025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.640032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.663016] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.678025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.691037] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.712025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.732731] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.744081] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.756025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.772022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.799961] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.808024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.815021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.834027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.842313] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.847025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.854028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.871022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.881517] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.885466] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.887394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.889448] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.892273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.897022] iommu: Default domain type: Passthrough [ 0.898404] SCSI subsystem initialized [ 0.899142] ACPI: bus type USB registered [ 0.901148] usbcore: registered new interface driver usbfs [ 0.902078] usbcore: registered new interface driver hub [ 0.904093] usbcore: registered new device driver usb [ 0.913184] pps_core: LinuxPPS API ver. 1 registered [ 0.915015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.918071] PTP clock support registered [ 0.920132] EDAC MC: Ver: 3.0.0 [ 0.922550] PCI: Using ACPI for IRQ routing [ 0.924042] NetLabel: Initializing [ 0.926011] NetLabel: domain hash size = 128 [ 0.927012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.929090] NetLabel: unlabeled traffic allowed by default [ 0.932096] vgaarb: loaded [ 0.933299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.935017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.941047] clocksource: Switched to clocksource kvm-clock [ 1.057680] VFS: Disk quotas dquot_6.6.0 [ 1.059360] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.061310] *** VALIDATE ramfs *** [ 1.062103] *** VALIDATE hugetlbfs *** [ 1.063291] pnp: PnP ACPI init [ 1.065547] pnp: PnP ACPI: found 6 devices [ 1.081426] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.085106] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.087439] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.089847] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.092383] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.095078] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.098326] NET: Registered protocol family 2 [ 1.100941] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.105372] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.108320] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.113705] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.117181] TCP: Hash tables configured (established 65536 bind 65536) [ 1.120091] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.123189] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.125912] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.128934] NET: Registered protocol family 1 [ 1.131401] RPC: Registered named UNIX socket transport module. [ 1.134144] RPC: Registered udp transport module. [ 1.135783] RPC: Registered tcp transport module. [ 1.136971] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.139178] NET: Registered protocol family 44 [ 1.140996] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.142839] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.144658] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.146701] PCI: CLS 0 bytes, default 64 [ 1.148196] Unpacking initramfs... [ 2.510450] debug: unmapping init [mem 0xffff8d317cc54000-0xffff8d317ffbffff] [ 2.514258] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.516735] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.519482] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.977485] Initialise system trusted keyrings [ 2.979137] Key type blacklist registered [ 2.981206] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.989943] zbud: loaded [ 2.994749] *** VALIDATE nfs *** [ 2.996127] *** VALIDATE nfs4 *** [ 2.997812] pstore: using deflate compression [ 3.000621] Platform Keyring initialized [ 3.079439] NET: Registered protocol family 38 [ 3.081755] Key type asymmetric registered [ 3.083047] Asymmetric key parser 'x509' registered [ 3.084234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.086504] io scheduler mq-deadline registered [ 3.087449] io scheduler kyber registered [ 3.088508] io scheduler bfq registered [ 3.089628] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.091410] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.093188] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.094912] ACPI: Power Button [PWRF] [ 3.098645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.102768] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.110414] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.116362] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.133846] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.159451] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.187250] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.191077] Non-volatile memory driver v1.3 [ 3.192176] Linux agpgart interface v0.103 [ 3.216367] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.218268] vda: detected capacity change from 0 to 69001216 [ 3.232649] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.234427] vdb: detected capacity change from 0 to 1073741824 [ 3.247887] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.250374] vdc: detected capacity change from 0 to 2621440000 [ 3.265545] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.267537] vdd: detected capacity change from 0 to 2621440000 [ 3.283224] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.285169] vde: detected capacity change from 0 to 4294967296 [ 3.297355] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.300017] vdf: detected capacity change from 0 to 4294967296 [ 3.307245] libphy: Fixed MDIO Bus: probed [ 3.316192] usbcore: registered new interface driver usbserial_generic [ 3.318937] usbserial: USB Serial support registered for generic [ 3.321698] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.326366] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.328352] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.331094] mousedev: PS/2 mouse device common for all mice [ 3.333800] rtc_cmos 00:05: RTC can wake from S4 [ 3.336263] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.340172] rtc_cmos 00:05: registered as rtc0 [ 3.342744] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.345260] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.352082] intel_pstate: CPU model not supported [ 3.353505] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.358797] hid: raw HID events driver (C) Jiri Kosina [ 3.360902] usbcore: registered new interface driver usbhid [ 3.362553] usbhid: USB HID core driver [ 3.364163] drop_monitor: Initializing network drop monitor service [ 3.366841] Initializing XFRM netlink socket [ 3.368942] NET: Registered protocol family 10 [ 3.372172] Segment Routing with IPv6 [ 3.373482] NET: Registered protocol family 17 [ 3.375423] mpls_gso: MPLS GSO support [ 3.381768] RAS: Correctable Errors collector initialized. [ 3.383422] AVX version of gcm_enc/dec engaged. [ 3.385010] AES CTR mode by8 optimization enabled [ 3.455843] sched_clock: Marking stable (3455815608, 0)->(4431278547, -975462939) [ 3.459497] registered taskstats version 1 [ 3.461304] Loading compiled-in X.509 certificates [ 3.463507] zswap: loaded using pool lzo/zbud [ 3.485635] Key type big_key registered [ 3.498679] Key type encrypted registered [ 3.500232] ima: No TPM chip found, activating TPM-bypass! [ 3.502370] ima: Allocated hash algorithm: sha1 [ 3.504285] ima: No architecture policies found [ 3.506089] evm: Initialising EVM extended attributes: [ 3.508163] evm: security.selinux [ 3.509097] evm: security.ima [ 3.509841] evm: security.capability [ 3.511136] evm: HMAC attrs: 0x1 [ 3.513404] rtc_cmos 00:05: setting system clock to 2026-05-25 05:41:09 UTC (1779687669) [ 3.519374] debug: unmapping init [mem 0xffffffffbd203000-0xffffffffbd3fffff] [ 3.522363] debug: unmapping init [mem 0xffffffffbbf82000-0xffffffffbc258fff] [ 3.532081] Write protecting the kernel read-only data: 28672k [ 3.535528] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 3.538442] debug: unmapping init [mem 0xffffffffbaf14000-0xffffffffbaffffff] [ 3.573193] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.581751] systemd[1]: Detected virtualization kvm. [ 3.584258] systemd[1]: Detected architecture x86-64. [ 3.585803] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.611296] systemd[1]: No hostname configured. [ 3.612422] systemd[1]: Set hostname to . [ 3.613844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.615570] systemd[1]: Initializing machine ID from random generator. [ 3.742818] random: systemd: uninitialized urandom read (16 bytes read) [ 3.745671] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.749359] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752355] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.756815] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.362103] device-mapper: uevent: version 1.0.3 [ 4.364482] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.106041] random: fast init done [ 5.123813] virtio_net virtio0 ens2: renamed from eth0 [ 5.178499] scsi host0: ata_piix [ 5.221763] scsi host1: ata_piix [ 5.223247] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.225864] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.030334] random: crng init done [ 10.035954] random: 7 urandom warning(s) missed due to ratelimiting [ 10.963452] dracut-initqueue[591]: RTNETLINK answers: File exists [ 11.666131] hrtimer: interrupt took 20101400 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.465874] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.296327] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.194799] SELinux: Disabled at runtime. [ 18.404632] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.427166] systemd[1]: Detected virtualization kvm. [ 18.430284] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.594893] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.600240] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.613954] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.622211] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.627955] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.644182] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.657105] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ 20.228604] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.338863] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.737698] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.928788] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.298681] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.388709] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 28.606442] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 29.635760] NFS: Registering the id_resolver key type [ 29.639138] Key type id_resolver registered [ 29.641487] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg256-server login: [ 98.707878] libcfs: loading out-of-tree module taints kernel. [ 98.803184] Key type ._llcrypt registered [ 98.804652] Key type .llcrypt registered [ 98.945575] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 124.343979] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 126.515414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.530675] alg: No test for adler32 (adler32-zlib) [ 128.336447] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 129.463194] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 131.327151] Key type lgssc registered [ 133.586472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 153.977559] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 205.501566] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 219.810943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 219.859435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.179971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.206981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.297461] Lustre: lustre-MDT0000: new disk, initializing [ 221.371255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.387249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 225.970739] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.111970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.242841] Lustre: 6513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.271508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.274210] Lustre: Skipped 1 previous similar message [ 239.431444] Lustre: lustre-MDT0001: new disk, initializing [ 239.526468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.560177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.589423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 244.721578] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.847347] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.344218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.602566] Lustre: lustre-OST0000: new disk, initializing [ 260.606227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.616899] Lustre: 8417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.665503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.310109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 266.327530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 266.516109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 267.895391] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.634381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.820076] Lustre: lustre-OST0001: new disk, initializing [ 284.826183] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.841206] Lustre: 9473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.963847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 290.326887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 290.340204] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 290.423629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 291.153498] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 304.354978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.695462] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 322.848760] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.748756] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing yml_node [ 334.013549] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 336.990890] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 340.165600] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 342.189660] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon May 25 01:46:45 EDT 2026 [ 363.467482] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 365.383403] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 382.944636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 382.960186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.986537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.554204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.577131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.235308] Lustre: server umount lustre-MDT0000 complete [ 393.634149] LustreError: 10328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688059 with bad export cookie 1860328902804892695 [ 393.651799] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.654967] LustreError: 10328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 397.796503] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.798683] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.838933] Lustre: Skipped 2 previous similar messages [ 397.878585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 397.879512] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 397.910303] Lustre: Skipped 4 previous similar messages [ 398.818878] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.851048] LustreError: 6519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 400.060450] Lustre: server umount lustre-MDT0001 complete [ 405.082738] Lustre: server umount lustre-OST0000 complete [ 409.013612] Lustre: server umount lustre-OST0001 complete [ 416.868319] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 427.117982] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 469.224702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.767243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 469.846086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 469.971532] Lustre: lustre-MDT0000: new disk, initializing [ 470.101263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.141931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 477.371176] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 490.551320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.663534] Lustre: 14990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 490.773244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 490.784771] Lustre: Skipped 1 previous similar message [ 490.958742] Lustre: lustre-MDT0001: new disk, initializing [ 491.107670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 491.141583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 491.156855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 496.571094] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 504.636382] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 509.093672] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 518.667978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.043971] Lustre: lustre-OST0000: new disk, initializing [ 519.065410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 519.082910] Lustre: 16734:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 519.248941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.599027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 520.615761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 520.741861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 526.592801] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 536.287536] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 541.185959] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.618455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 546.861989] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 547.138890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 551.392491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.398993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 551.407271] Lustre: Skipped 1 previous similar message [ 551.411069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.429198] Lustre: Skipped 3 previous similar messages [ 556.536247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.545467] Lustre: Skipped 1 previous similar message [ 563.167143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 563.458609] Lustre: server umount lustre-OST0000 complete [ 567.781122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 567.792679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.812455] Lustre: Skipped 1 previous similar message [ 567.820564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.826573] Lustre: Skipped 2 previous similar messages [ 573.924243] Lustre: server umount lustre-MDT0000 complete [ 576.997471] LustreError: 15853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.025370] LustreError: 15853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 578.286144] LustreError: 14982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688244 with bad export cookie 1860328902804894228 [ 578.287638] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.304947] LustreError: 14982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 578.762644] Lustre: server umount lustre-MDT0001 complete [ 583.176314] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:50:46 (1779688246) [ 585.721231] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 587.755762] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:50:51 (1779688251) [ 594.799305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.405303] LustreError: 18998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.465376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.216461] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 600.552197] LustreError: 18999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.633819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.807598] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 621.696426] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 626.603766] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 634.373417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 642.247436] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 647.580634] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 656.359803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 656.371391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 656.400110] Lustre: Skipped 1 previous similar message [ 656.418219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 656.428946] Lustre: Skipped 2 previous similar messages [ 661.097940] Lustre: server umount lustre-OST0000 complete [ 666.081894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.086051] LustreError: Skipped 1 previous similar message [ 671.469792] Lustre: server umount lustre-MDT0000 complete [ 672.738278] LustreError: 20907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.772914] LustreError: 20907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 675.999286] LustreError: 18978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688341 with bad export cookie 1860328902804895110 [ 676.015238] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.015695] LustreError: 18978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 676.428329] Lustre: server umount lustre-MDT0001 complete [ 688.194453] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 691.634841] Key type lgssc unregistered [ 692.159941] LNet: 22176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.171684] LNetError: 22176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.227270] LNet: Removed LNI 192.168.202.156@tcp [ 694.200046] Key type .llcrypt unregistered [ 694.201111] Key type ._llcrypt unregistered [ 717.690393] Key type ._llcrypt registered [ 717.692704] Key type .llcrypt registered [ 717.843734] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 719.177925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 719.340083] alg: No test for adler32 (adler32-zlib) [ 720.481731] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 720.794769] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 722.576319] Key type lgssc registered [ 723.713713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.229587] loop: module loaded [ 732.284146] loop0: detected capacity change from 0 to 204800000 [ 732.562238] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.572122] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.954510] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.586854] loop0: detected capacity change from 0 to 204800000 [ 738.782869] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.802381] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.823661] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 744.588926] loop0: detected capacity change from 0 to 204800000 [ 750.978766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 750.998852] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.338228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 752.434984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 752.548361] Lustre: lustre-MDT0000: new disk, initializing [ 752.658467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.696929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 756.550777] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 765.396045] loop1: detected capacity change from 0 to 204800000 [ 771.729121] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.227268] Lustre: lustre-OST07c6: new disk, initializing [ 772.231727] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 772.236740] Lustre: Skipped 1 previous similar message [ 772.240779] Lustre: 25329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 772.327715] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 778.996556] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 780.395954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 780.405961] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 780.444485] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 784.301391] Lustre: 24333:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 784.308461] Lustre: 24333:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 784.312750] Lustre: 24333:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 784.317474] Lustre: 24333:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 784.323227] Lustre: 24333:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 784.327471] Lustre: 24333:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 799.353551] Lustre: 24334:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 799.368337] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.374355] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.380101] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 799.386286] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 799.395984] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 804.674322] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 804.687310] Lustre: 24335:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 804.699140] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 804.710826] Lustre: 24335:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.721561] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 804.735961] Lustre: 24335:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.748836] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 804.773737] Lustre: 24335:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.788497] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 804.804808] Lustre: 24335:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.822813] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 804.838791] Lustre: 24335:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.085149] Lustre: 24334:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 808.106569] Lustre: 24334:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 808.113382] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 808.119304] Lustre: 24334:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.129040] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.134719] Lustre: 24334:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.139809] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 808.145822] Lustre: 24334:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.154732] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 808.163131] Lustre: 24334:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.170851] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.177245] Lustre: 24334:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.320686] Lustre: 25977:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 813.330148] Lustre: 25977:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.341403] Lustre: 25977:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 813.355987] Lustre: 25977:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.364596] Lustre: 25977:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.374926] Lustre: 25977:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.385204] Lustre: 25977:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 813.393923] Lustre: 25977:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.402838] Lustre: 25977:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 813.411733] Lustre: 25977:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.427794] Lustre: 25977:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.435651] Lustre: 25977:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.471355] Lustre: 25977:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 824.476248] Lustre: 25977:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 824.480924] Lustre: 25977:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 824.486759] Lustre: 25977:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 824.497746] Lustre: 25977:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.503879] Lustre: 25977:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 824.519696] Lustre: 25977:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 824.532570] Lustre: 25977:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 824.542103] Lustre: 25977:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 824.552127] Lustre: 25977:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 824.573260] Lustre: 25977:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 824.585520] Lustre: 25977:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 877.552574] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.567986] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 880.422473] Lustre: server umount lustre-OST07c6 complete [ 891.659603] Lustre: server umount lustre-MDT0000 complete [ 908.355598] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 911.046605] Key type lgssc unregistered [ 911.308781] LNet: 27852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.323448] LNetError: 27852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.345242] LNet: Removed LNI 192.168.202.156@tcp [ 912.259181] Key type .llcrypt unregistered [ 912.260815] Key type ._llcrypt unregistered [ 934.016835] Key type ._llcrypt registered [ 934.021488] Key type .llcrypt registered [ 934.141291] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 935.861357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.953635] alg: No test for adler32 (adler32-zlib) [ 937.214255] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 937.566739] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 939.295567] Key type lgssc registered [ 940.773750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.364332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.385834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.026918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.954168] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 960.482585] LustreError: 29117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.499824] LustreError: 29117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 964.638551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 964.912930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 969.982836] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 975.958061] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.394397] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 987.471622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 988.068186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.666845] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 995.268489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1000.781890] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1019.872295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.883813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.902751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.898596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1020.912532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1022.607579] Lustre: server umount lustre-OST0000 complete [ 1031.139942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.141977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.152206] Lustre: Skipped 1 previous similar message [ 1031.176172] Lustre: Skipped 2 previous similar messages [ 1032.946816] Lustre: server umount lustre-MDT0000 complete [ 1036.287781] LustreError: 29118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.305483] LustreError: 29118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1037.035815] LustreError: 29835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688703 with bad export cookie 17510789494203892197 [ 1037.045337] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.050666] LustreError: 29835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1037.324341] Lustre: server umount lustre-MDT0001 complete [ 1047.908930] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1051.013774] Key type lgssc unregistered [ 1051.289557] LNet: 32493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.301824] LNetError: 32493:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.321299] LNet: Removed LNI 192.168.202.156@tcp [ 1052.157851] Key type .llcrypt unregistered [ 1052.161840] Key type ._llcrypt unregistered [ 1072.963487] Key type ._llcrypt registered [ 1072.965239] Key type .llcrypt registered [ 1073.116524] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1074.374539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1074.522877] alg: No test for adler32 (adler32-zlib) [ 1075.711239] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1076.072741] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1077.895178] Key type lgssc registered [ 1079.407157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1088.545369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1088.579049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.081217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.323382] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1100.266095] LustreError: 33746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.281938] LustreError: 33746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1104.960298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.376256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1105.515220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1110.077890] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1115.656689] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1119.588523] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1126.708284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1127.047428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.036773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1135.310230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1139.639304] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1150.949818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.964324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.968960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1150.976759] Lustre: Skipped 1 previous similar message [ 1150.990892] Lustre: Skipped 1 previous similar message [ 1156.070218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1156.090231] Lustre: Skipped 1 previous similar message [ 1156.374095] Lustre: server umount lustre-OST0000 complete [ 1161.185420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.186940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.196014] Lustre: Skipped 1 previous similar message [ 1161.207049] Lustre: Skipped 1 previous similar message [ 1166.307518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1166.317470] Lustre: Skipped 1 previous similar message [ 1166.518167] Lustre: server umount lustre-MDT0000 complete [ 1170.428558] LustreError: 35986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688836 with bad export cookie 2929948635272653811 [ 1170.442088] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.446295] LustreError: 35986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1170.933144] Lustre: server umount lustre-MDT0001 complete [ 1183.371844] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1186.441987] Key type lgssc unregistered [ 1186.724981] LNet: 36924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.732922] LNetError: 36924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.748472] LNet: Removed LNI 192.168.202.156@tcp [ 1187.569135] Key type .llcrypt unregistered [ 1187.571297] Key type ._llcrypt unregistered [ 1213.789956] Key type ._llcrypt registered [ 1213.792159] Key type .llcrypt registered [ 1213.914910] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1215.458400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1215.483143] alg: No test for adler32 (adler32-zlib) [ 1216.649896] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1216.871451] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1218.559609] Key type lgssc registered [ 1219.959581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1226.544780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1226.559763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.897024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.117074] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1237.479901] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1237.515881] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1242.594530] LustreError: 38348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.488257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.067575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1249.227340] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1254.366294] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.301446] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1263.914000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1264.357636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.591718] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1272.557390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1275.837932] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1282.777929] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1283.043846] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.056164] Lustre: Skipped 1 previous similar message [ 1287.874394] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1287.882457] Lustre: Skipped 3 previous similar messages [ 1287.991710] Lustre: server umount lustre-MDT0000 complete [ 1288.162733] LustreError: 38348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.184858] LustreError: 38348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1291.828907] LustreError: 38329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779688957 with bad export cookie 14546129051785664923 [ 1291.837100] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.840581] LustreError: 38329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1293.004164] LustreError: 38343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.020982] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 1293.285943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.299286] Lustre: Skipped 1 previous similar message [ 1296.291536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1296.294602] Lustre: Skipped 1 previous similar message [ 1298.112559] LustreError: 38344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.127145] LustreError: 38344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1298.347879] Lustre: server umount lustre-MDT0001 complete [ 1319.117275] Lustre: server umount lustre-OST0000 complete [ 1332.758622] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1335.518889] Key type lgssc unregistered [ 1335.802896] LNet: 41852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.808652] LNetError: 41852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.817741] LNet: Removed LNI 192.168.202.156@tcp [ 1336.411288] Key type .llcrypt unregistered [ 1336.413850] Key type ._llcrypt unregistered [ 1359.010584] Key type ._llcrypt registered [ 1359.013276] Key type .llcrypt registered [ 1359.143672] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1360.179942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1360.216633] alg: No test for adler32 (adler32-zlib) [ 1361.293753] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1361.604825] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1363.471278] Key type lgssc registered [ 1364.927518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1373.747665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.770873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.221931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.782701] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1385.442070] LustreError: 43253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1385.450785] LustreError: 43253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1386.973918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.447055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1394.381416] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1400.555806] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1405.425581] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1413.106926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.648765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.702058] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1420.857198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1426.096793] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1436.141983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.147177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.154973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.155239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.155862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1437.172919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1438.958309] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1438.965542] Lustre: Skipped 1 previous similar message [ 1442.279454] LustreError: 44847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.307930] LustreError: 44847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1442.548016] Lustre: server umount lustre-OST0000 complete [ 1462.771647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.789259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1462.790429] Lustre: Skipped 1 previous similar message [ 1462.819805] Lustre: Skipped 1 previous similar message [ 1467.358492] Lustre: server umount lustre-MDT0000 complete [ 1467.872814] LustreError: 43254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.888738] LustreError: 43254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1471.053438] LustreError: 43235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689137 with bad export cookie 7599025869213704661 [ 1471.063399] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1471.067011] LustreError: 43235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1477.566512] Lustre: server umount lustre-MDT0001 complete [ 1489.394591] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1492.482185] Key type lgssc unregistered [ 1492.739930] LNet: 46606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.746968] LNetError: 46606:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.758466] LNet: Removed LNI 192.168.202.156@tcp [ 1493.634233] Key type .llcrypt unregistered [ 1493.636114] Key type ._llcrypt unregistered [ 1518.101397] Key type ._llcrypt registered [ 1518.103373] Key type .llcrypt registered [ 1518.203563] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1519.748308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.806453] alg: No test for adler32 (adler32-zlib) [ 1521.063755] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1521.441424] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1523.231368] Key type lgssc registered [ 1524.539131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1533.048477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1533.061159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1534.589657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.287900] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1544.167670] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.201755] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1548.462404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.830933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1553.491527] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1560.474571] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1565.184736] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1572.333505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.781177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.134146] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1581.000437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1586.337937] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1592.838432] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1596.182558] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1608.299278] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-25 2h07m51s [ 1609.725915] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1611.750484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.757300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.759164] Lustre: Skipped 2 previous similar messages [ 1611.771734] Lustre: Skipped 2 previous similar messages [ 1612.835365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.844590] Lustre: Skipped 2 previous similar messages [ 1614.551293] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 1617.895787] Lustre: server umount lustre-MDT0000 complete [ 1618.918839] LustreError: 48976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.934562] LustreError: 48976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1620.675584] LustreError: 48002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.776560] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1625.063205] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.086021] LustreError: 48006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1629.119330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.340713] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.692249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.748671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1630.709739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1634.048901] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1637.727734] Lustre: DEBUG MARKER: Wait for df (36100) ... [ 1639.467703] Lustre: DEBUG MARKER: done [ 1645.024713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1645.039270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1645.058157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.070573] Lustre: Skipped 1 previous similar message [ 1645.551663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1650.416763] Lustre: server umount lustre-OST0000 complete [ 1656.290179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.301489] Lustre: Skipped 1 previous similar message [ 1656.312734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.320644] Lustre: Skipped 3 previous similar messages [ 1660.764262] Lustre: server umount lustre-MDT0000 complete [ 1663.469962] LustreError: 48976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.487434] LustreError: 48976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1664.765681] LustreError: 50296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689330 with bad export cookie 8696512282619811350 [ 1664.766902] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1664.774803] LustreError: 50296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1665.110663] Lustre: server umount lustre-MDT0001 complete [ 1677.369158] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1680.090638] Key type lgssc unregistered [ 1680.412755] LNet: 52557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.420725] LNetError: 52557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.441892] LNet: Removed LNI 192.168.202.156@tcp [ 1681.227262] Key type .llcrypt unregistered [ 1681.234959] Key type ._llcrypt unregistered [ 1728.958976] Key type ._llcrypt registered [ 1728.970083] Key type .llcrypt registered [ 1729.141612] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1731.338684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1731.348724] alg: No test for adler32 (adler32-zlib) [ 1732.574563] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1732.819565] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1734.527560] Key type lgssc registered [ 1736.455912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1744.838889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1744.864470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.072107] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1746.081554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1746.195122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.660829] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1758.783717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.863759] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1758.896021] Lustre: 54752:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1759.118520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1764.208953] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1770.269420] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.424447] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1781.270456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1781.468646] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1781.707828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.606486] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1791.655688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1794.315232] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1801.228614] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-25 2h11m04s [ 1803.431678] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1811.686328] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1814.321236] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1814.373862] LustreError: 54759:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1834.725788] Lustre: lustre-MDT0000: Client fddc5e03-8cd6-4ca3-be88-4cbe70a225b4 (at 192.168.202.56@tcp) reconnecting [ 1854.383780] LustreError: 54759:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1856.682854] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1859.135772] Lustre: DEBUG MARKER: done [ 1867.702624] Lustre: server umount lustre-OST0000 complete [ 1871.840714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1871.855203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.866534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.928242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.932966] Lustre: Skipped 2 previous similar messages [ 1879.010890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.014358] Lustre: Skipped 1 previous similar message [ 1881.153489] Lustre: server umount lustre-MDT0000 complete [ 1884.131826] Lustre: 54074:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689534/real 1779689534] req@000000007a011fd9 x1866139590251904/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779689550 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1884.165841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.185693] Lustre: Skipped 1 previous similar message [ 1884.847885] LustreError: 54744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689550 with bad export cookie 16294506356482499536 [ 1884.850838] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1884.854663] LustreError: 54744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1885.227292] Lustre: server umount lustre-MDT0001 complete [ 1895.661440] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 1897.979817] Key type lgssc unregistered [ 1898.172677] LNet: 58725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.177308] LNetError: 58725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.189990] LNet: Removed LNI 192.168.202.156@tcp [ 1899.014341] Key type .llcrypt unregistered [ 1899.017136] Key type ._llcrypt unregistered [ 1947.644516] Key type ._llcrypt registered [ 1947.646707] Key type .llcrypt registered [ 1947.738444] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 1949.111671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.149786] alg: No test for adler32 (adler32-zlib) [ 1950.326482] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 1950.561947] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 1952.223844] Key type lgssc registered [ 1953.401319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1956.259384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1956.269393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.485668] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1957.503719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1957.633550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1958.730700] Lustre: Failing over lustre-MDT0000 [ 1959.131816] Lustre: server umount lustre-MDT0000 complete [ 1966.150727] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:13:49 (1779689629) [ 1972.941042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.479836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.934408] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1988.007837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.225316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1988.289930] Lustre: 61508:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1988.670792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.562906] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 1999.903200] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2004.323865] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2011.531149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.793134] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2012.054468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.499300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2019.280333] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2025.317344] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2031.509620] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2039.268504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.271219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.285338] Lustre: Skipped 2 previous similar messages [ 2039.300021] Lustre: Skipped 2 previous similar messages [ 2043.599111] Lustre: server umount lustre-MDT0000 complete [ 2047.601940] LustreError: 62228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689713 with bad export cookie 13521376275456019146 [ 2047.608852] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.614472] LustreError: 62228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2047.888075] Lustre: server umount lustre-MDT0001 complete [ 2051.807937] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2056.356037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2064.690517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.889757] Lustre: 60432:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689715/real 1779689715] req@0000000073a3443b x1866139818827008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779689731 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.935268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.959173] Lustre: 60430:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689718/real 1779689718] req@00000000912674e5 x1866139818827136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779689734 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.050422] Lustre: 60430:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689720/real 1779689720] req@0000000032054c10 x1866139818827264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779689736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.095689] LustreError: 60429:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000a62c9c1 x1866139818828032/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.459842] LustreError: 63131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.484757] LustreError: 63131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2072.604932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.710881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2073.567130] Lustre: 60432:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689723/real 1779689723] req@000000001e7d4d77 x1866139818827520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779689739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2074.669091] LustreError: 63132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.697509] LustreError: 64937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.715792] LustreError: 64937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2076.117777] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2078.753896] Lustre: 60432:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689728/real 1779689728] req@00000000e7c7f49a x1866139818827648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779689744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2080.737960] LustreError: 63132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.759873] LustreError: 63132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2084.499179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.980701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.262786] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2090.467392] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2094.581480] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2094.887343] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.583113] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2109.926712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.929197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2109.944640] Lustre: Skipped 2 previous similar messages [ 2109.960556] Lustre: Skipped 2 previous similar messages [ 2111.873144] Lustre: server umount lustre-MDT0000 complete [ 2115.044698] LustreError: 64937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.069085] LustreError: 64937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2115.195182] LustreError: 62228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689781 with bad export cookie 13521376275456021421 [ 2115.197608] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2115.211695] LustreError: 62228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2115.763099] Lustre: server umount lustre-MDT0001 complete [ 2122.225140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2132.138194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.544049] Lustre: 60433:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779689786/real 1779689786] req@000000008d3181b1 x1866139818853504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779689802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.574556] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.715478] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbba594154cf35910 [ 2140.727342] Lustre: MGC192.168.202.156@tcp: Connection restored to 0@lo (at 0@lo) [ 2141.098574] LustreError: 63131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.171745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.228768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2145.126288] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2153.697264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.509916] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2159.078539] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2164.390741] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2165.312909] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2167.852653] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2174.438600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2174.451125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.465907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.472552] LustreError: 63132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.482653] LustreError: 63132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2175.587627] Lustre: server umount lustre-MDT0000 complete [ 2179.205767] LustreError: 61501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779689845 with bad export cookie 13521376275456022800 [ 2179.210466] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.215436] LustreError: 61501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2179.602721] Lustre: server umount lustre-MDT0001 complete [ 2186.289478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2189.438514] Lustre: DEBUG MARKER: files compared the same [ 2201.425810] Lustre: server umount lustre-OST0000 complete [ 2215.109341] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2217.705145] Key type lgssc unregistered [ 2218.018422] LNet: 70425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.028393] LNetError: 70425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.043711] LNet: Removed LNI 192.168.202.156@tcp [ 2218.870048] Key type .llcrypt unregistered [ 2218.871981] Key type ._llcrypt unregistered [ 2240.153484] Key type ._llcrypt registered [ 2240.155638] Key type .llcrypt registered [ 2240.256984] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2241.420466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.600310] alg: No test for adler32 (adler32-zlib) [ 2242.749618] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2242.998468] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2244.743611] Key type lgssc registered [ 2245.766761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.968865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.982943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.399431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.009572] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2264.361365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.593296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2268.237763] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2272.649673] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2276.031648] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2281.410060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2281.630792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.390284] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug +malloc all [ 2292.945549] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2300.088378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2315.744063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.745628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.746787] LustreError: Skipped 1 previous similar message [ 2315.748948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.772479] Lustre: Skipped 1 previous similar message [ 2317.023466] Lustre: server umount lustre-OST0000 complete [ 2320.863812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.863849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.867405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.867414] Lustre: Skipped 1 previous similar message [ 2320.881327] Lustre: Skipped 1 previous similar message [ 2325.984345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2325.988934] Lustre: Skipped 2 previous similar messages [ 2331.104292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.107241] Lustre: Skipped 1 previous similar message [ 2334.175190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2335.201794] LustreError: 72533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.221506] LustreError: 72533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2336.374505] Lustre: server umount lustre-MDT0000 complete [ 2339.307769] LustreError: 71803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690005 with bad export cookie 17083109605247943467 [ 2339.309862] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2339.315681] LustreError: 71803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2339.558813] Lustre: server umount lustre-MDT0001 complete [ 2349.158925] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2351.234359] Key type lgssc unregistered [ 2351.479424] LNet: 75021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.489655] LNetError: 75021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.518312] LNet: Removed LNI 192.168.202.156@tcp [ 2352.265179] Key type .llcrypt unregistered [ 2352.267472] Key type ._llcrypt unregistered [ 2372.605741] Key type ._llcrypt registered [ 2372.607701] Key type .llcrypt registered [ 2372.686161] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2373.689252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.764479] alg: No test for adler32 (adler32-zlib) [ 2374.762457] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2374.925103] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2376.559177] Key type lgssc registered [ 2377.698966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2384.347433] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2384.375200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.903409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2405.023255] LustreError: 76394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [ 2431.711192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.588724] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2441.169826] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2447.609533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.726366] Lustre: *** cfs_fail_loc=706, val=0*** [ 2448.020857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.095722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2451.699307] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2453.482437] LustreError: 76419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.517744] LustreError: 76419:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2458.592968] LustreError: 76420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.833609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.220939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.139195] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2466.961284] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2469.562947] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2471.399393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.404657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.411793] Lustre: Skipped 1 previous similar message [ 2471.430819] Lustre: Skipped 1 previous similar message [ 2476.515285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.524821] Lustre: Skipped 1 previous similar message [ 2481.641637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.651712] Lustre: Skipped 1 previous similar message [ 2485.215187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2485.361566] Lustre: server umount lustre-OST0000 complete [ 2489.824396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2489.832979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.857912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2491.876586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.495588] Lustre: server umount lustre-MDT0000 complete [ 2496.992670] LustreError: 77371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.003747] LustreError: 77371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2497.316915] LustreError: 76400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690163 with bad export cookie 9847313090260558491 [ 2497.325178] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2497.624619] Lustre: server umount lustre-MDT0001 complete [ 2505.959953] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2508.079488] Key type lgssc unregistered [ 2508.280856] LNet: 79653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.283301] LNetError: 79653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2508.293996] LNet: Removed LNI 192.168.202.156@tcp [ 2509.043627] Key type .llcrypt unregistered [ 2509.045824] Key type ._llcrypt unregistered [ 2528.325466] Key type ._llcrypt registered [ 2528.330099] Key type .llcrypt registered [ 2528.449611] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2529.186072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2529.208077] alg: No test for adler32 (adler32-zlib) [ 2530.268315] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2530.475597] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2532.145142] Key type lgssc registered [ 2533.290375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.547850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2539.574304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.102671] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2550.092213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2550.480836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.583725] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2555.873949] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.895707] LustreError: 81702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2560.211223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2560.500698] LustreError: 81700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.517579] LustreError: 81700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2560.534713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.575147] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.598039] LustreError: 81705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2565.561136] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2566.623961] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.649433] LustreError: 81723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2571.753918] LustreError: 82431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.778834] LustreError: 82431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2572.543909] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.565885] Lustre: MGS: Client 90f9c001-42b0-4148-ad01-5fb3d0ee4b77 (at 0@lo) reconnecting [ 2572.594476] LustreError: 81716:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2572.775287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.820198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2576.517425] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2588.136019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2588.145674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2588.165285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.268058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.272112] Lustre: Skipped 1 previous similar message [ 2593.996315] Lustre: server umount lustre-OST0000 complete [ 2598.369964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2598.370745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.372275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.372280] Lustre: Skipped 1 previous similar message [ 2598.396170] Lustre: Skipped 2 previous similar messages [ 2603.493083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2603.498573] Lustre: Skipped 2 previous similar messages [ 2603.741617] Lustre: server umount lustre-MDT0000 complete [ 2607.045937] Lustre: server umount lustre-MDT0001 complete [ 2609.921763] Lustre: server umount MGS complete [ 2619.343994] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:24:43 (1779690283) [ 2644.394357] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 2652.605487] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2680.661191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.408167] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2691.209200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.398889] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2691.451127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2691.514365] Lustre: lustre-MDT0001: new disk, initializing [ 2691.579957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.600408] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2691.612307] Lustre: 88765:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2691.626826] LustreError: 88765:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2694.702451] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2703.079090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2703.275828] Lustre: lustre-OST0000: new disk, initializing [ 2703.279458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2703.284513] Lustre: 89512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2703.333230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.275475] Lustre: 89642:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c3b3a926 x1866140426999552/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779690388 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2708.340438] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2714.731993] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2720.706561] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2720.724380] Lustre: MGS: Client 66a65b82-c6d4-440a-95b4-36e5008b5348 (at 0@lo) reconnecting [ 2720.822841] Lustre: 88051:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.839353] LustreError: 88751:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2720.881387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2720.936277] Lustre: lustre-MDT0000: new disk, initializing [ 2721.023602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.037832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2721.059464] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2722.080934] Lustre: 90470:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008a3ad4a1 x1866140427006592/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1779690404 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2724.412840] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2726.892580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2726.900781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2727.207704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2747.364255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2747.372123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.387466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2753.433808] Lustre: server umount lustre-OST0000 complete [ 2757.089120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.089934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2757.092615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.092744] Lustre: Skipped 3 previous similar messages [ 2757.099534] Lustre: Skipped 2 previous similar messages [ 2757.115828] LustreError: Skipped 1 previous similar message [ 2762.711534] Lustre: server umount lustre-MDT0000 complete [ 2771.945057] Lustre: server umount lustre-MDT0001 complete [ 2774.857461] Lustre: server umount MGS complete [ 2782.836364] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:27:27 (1779690447) [ 2795.398538] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2797.459917] Key type lgssc unregistered [ 2797.656888] LNet: 92936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.661276] LNetError: 92936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.698099] LNet: Removed LNI 192.168.202.156@tcp [ 2799.296981] Key type .llcrypt unregistered [ 2799.298897] Key type ._llcrypt unregistered [ 2812.323932] Key type ._llcrypt registered [ 2812.328353] Key type .llcrypt registered [ 2812.430826] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2813.228651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.257835] alg: No test for adler32 (adler32-zlib) [ 2814.235988] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2814.379631] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2816.039363] Key type lgssc registered [ 2816.954716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2823.026472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2823.044239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.506831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.464693] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2834.915713] LustreError: 94121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.950934] LustreError: 94121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2835.873943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.210932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.057470] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2845.267617] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2848.297930] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2853.126397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2853.382075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.181789] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2862.102704] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2869.810799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2879.977604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.986551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.002767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2880.006961] Lustre: Skipped 1 previous similar message [ 2884.569565] Lustre: server umount lustre-OST0000 complete [ 2887.657848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2887.672063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.679433] Lustre: Skipped 1 previous similar message [ 2887.690863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2890.210501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.218480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2890.242117] Lustre: Skipped 1 previous similar message [ 2893.699308] Lustre: server umount lustre-MDT0000 complete [ 2895.332291] LustreError: 94117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.341963] LustreError: 94117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2896.465890] LustreError: 94102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690562 with bad export cookie 9457075156668312472 [ 2896.470167] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.479053] LustreError: 94102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2896.747335] Lustre: server umount lustre-MDT0001 complete [ 2904.758731] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 2906.977574] Key type lgssc unregistered [ 2907.234846] LNet: 97318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.239846] LNetError: 97318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.252154] LNet: Removed LNI 192.168.202.156@tcp [ 2907.895119] Key type .llcrypt unregistered [ 2907.896468] Key type ._llcrypt unregistered [ 2920.024202] Key type ._llcrypt registered [ 2920.027937] Key type .llcrypt registered [ 2920.120513] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 2921.118020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2921.147311] alg: No test for adler32 (adler32-zlib) [ 2922.104489] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2922.260981] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 2923.911198] Key type lgssc registered [ 2924.776147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2929.848227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2929.859789] LustreError: 98362:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2929.862727] LustreError: 98363:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2929.870929] LustreError: 98362:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2929.880050] LustreError: 98362:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2929.889598] LustreError: 98362:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2929.900754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.302970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.420641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.666764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2941.029192] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2945.140936] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lsmod [ 2947.241572] LustreError: 99534:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2947.245481] LustreError: 99533:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2947.253972] LustreError: 99534:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2947.263736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.264115] LustreError: 99534:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2947.264155] LustreError: 99534:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2947.591870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.189618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2953.199421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.211282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.700716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.723558] Lustre: Skipped 1 previous similar message [ 2953.736268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.747652] Lustre: Skipped 1 previous similar message [ 2956.128442] Lustre: server umount lustre-MDT0000 complete [ 2958.823719] LustreError: 98397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.836598] LustreError: 98397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2959.308198] LustreError: 98381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690625 with bad export cookie 211271070318873327 [ 2959.320878] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2959.324448] LustreError: 98381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2959.864051] LustreError: 98832:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e1677d97 x1866140838022016/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2960.041838] Lustre: server umount lustre-MDT0001 complete [ 2973.328516] Lustre: server umount lustre-OST0000 complete [ 2979.372504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.702309] LustreError: 100599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.744047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.737271] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2984.935914] LustreError: 100600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2988.561546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.778290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2992.000686] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 2996.223548] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2998.930869] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3004.236557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3004.461075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3009.022212] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3013.110798] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3024.941140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3034.080526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3034.084833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.094805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3035.111881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3038.556801] Lustre: server umount lustre-OST0000 complete [ 3045.344349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.345549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.356323] Lustre: Skipped 2 previous similar messages [ 3045.358702] Lustre: Skipped 2 previous similar messages [ 3047.437330] Lustre: server umount lustre-MDT0000 complete [ 3050.367974] LustreError: 100581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690716 with bad export cookie 211271070318874027 [ 3050.372148] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.377522] LustreError: 100581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3050.701703] Lustre: server umount lustre-MDT0001 complete [ 3059.365640] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3061.719701] Key type lgssc unregistered [ 3061.949625] LNet: 103793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3061.955392] LNetError: 103793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3063.020963] LNet: Removed LNI 192.168.202.156@tcp [ 3063.633612] Key type .llcrypt unregistered [ 3063.637079] Key type ._llcrypt unregistered [ 3080.416604] Key type ._llcrypt registered [ 3080.418409] Key type .llcrypt registered [ 3080.473602] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3081.046986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3081.104391] alg: No test for adler32 (adler32-zlib) [ 3082.120737] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3082.265956] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3083.903253] Key type lgssc registered [ 3084.727734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3089.602494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3089.615350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.900436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.622753] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3099.433426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.678921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3102.447432] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3107.162668] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3109.655473] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3114.453427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3114.673779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.697225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3118.533879] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3122.341423] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3129.908089] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3132.260467] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3135.457748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3135.465663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.472214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3135.483835] Lustre: Skipped 1 previous similar message [ 3140.581677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.589800] Lustre: Skipped 2 previous similar messages [ 3140.963581] Lustre: server umount lustre-OST0000 complete [ 3145.697703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.704958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3145.709458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.732147] Lustre: Skipped 1 previous similar message [ 3150.195414] Lustre: server umount lustre-MDT0000 complete [ 3150.819572] LustreError: 105171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.831355] LustreError: 105171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3152.962957] LustreError: 105153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690818 with bad export cookie 6225642709593257212 [ 3152.971393] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.971480] LustreError: 105153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3153.225437] Lustre: server umount lustre-MDT0001 complete [ 3160.257793] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3162.111255] Key type lgssc unregistered [ 3162.360519] LNet: 108448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.368920] LNetError: 108448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.387291] LNet: Removed LNI 192.168.202.156@tcp [ 3163.058628] Key type .llcrypt unregistered [ 3163.060154] Key type ._llcrypt unregistered [ 3175.642234] Key type ._llcrypt registered [ 3175.644997] Key type .llcrypt registered [ 3175.707724] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3176.490540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.578673] alg: No test for adler32 (adler32-zlib) [ 3177.597908] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3177.760734] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3179.383179] Key type lgssc registered [ 3180.286695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.571826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3186.589515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3187.928969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.229859] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3197.417575] LustreError: 109632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.432610] LustreError: 109632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3197.645293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.865921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.077954] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3205.699915] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3209.262784] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3214.662575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.881372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.044612] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3223.078512] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3232.292565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3242.466756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.477442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3242.478626] Lustre: Skipped 1 previous similar message [ 3242.483076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3242.486158] Lustre: Skipped 1 previous similar message [ 3245.115429] Lustre: server umount lustre-OST0000 complete [ 3247.584903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.585751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3247.593568] Lustre: Skipped 1 previous similar message [ 3252.703987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.706514] Lustre: Skipped 2 previous similar messages [ 3253.451462] Lustre: server umount lustre-MDT0000 complete [ 3255.731429] LustreError: 109613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779690921 with bad export cookie 973169993175938875 [ 3255.735779] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.741459] LustreError: 109613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3255.934173] Lustre: server umount lustre-MDT0001 complete [ 3263.658552] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3265.902856] Key type lgssc unregistered [ 3266.109679] LNet: 112830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3266.114571] LNetError: 112830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3266.124679] LNet: Removed LNI 192.168.202.156@tcp [ 3266.735054] Key type .llcrypt unregistered [ 3266.736347] Key type ._llcrypt unregistered [ 3282.577651] Key type ._llcrypt registered [ 3282.579504] Key type .llcrypt registered [ 3282.668298] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3283.250021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3283.274605] alg: No test for adler32 (adler32-zlib) [ 3284.277599] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3284.453027] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3286.095265] Key type lgssc registered [ 3287.005687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3292.551234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3292.576840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.915851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.107343] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3303.289894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.395352] LustreError: 114208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.412874] LustreError: 114208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3303.600640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3306.600392] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3310.500826] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3313.469913] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3319.087708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.328478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.188334] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3327.091718] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3338.806861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3352.934752] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3359.585250] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3359.846951] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3359.856209] Lustre: Skipped 2 previous similar messages [ 3368.158083] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3368.829154] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3368.833192] Lustre: Skipped 2 previous similar messages [ 3377.851264] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3377.859771] Lustre: Skipped 1 previous similar message [ 3387.022644] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3387.047775] Lustre: Skipped 2 previous similar messages [ 3395.270708] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.56@tcp 192.168.0.[2,10]@tcp [ 3402.631595] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.56/24@tcp in log lustre-MDT0000 [ 3402.640569] Lustre: Skipped 2 previous similar messages [ 3402.928620] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 3402.935300] Lustre: Skipped 1 previous similar message [ 3420.140941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3420.152508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.157945] LustreError: Skipped 1 previous similar message [ 3420.183463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.650360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.346934] Lustre: server umount lustre-OST0000 complete [ 3425.762169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.779698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3425.779708] Lustre: Skipped 1 previous similar message [ 3425.788899] Lustre: Skipped 1 previous similar message [ 3430.880381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.885411] Lustre: Skipped 2 previous similar messages [ 3432.054422] Lustre: server umount lustre-MDT0000 complete [ 3436.006354] LustreError: 116246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.037549] LustreError: 116246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3436.572186] LustreError: 114189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691102 with bad export cookie 3411884460566672193 [ 3436.573805] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.589544] LustreError: 114189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3437.367081] Lustre: server umount lustre-MDT0001 complete [ 3452.565789] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3456.112452] Key type lgssc unregistered [ 3456.453568] LNet: 118753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3456.459568] LNetError: 118753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3456.473875] LNet: Removed LNI 192.168.202.156@tcp [ 3457.742963] Key type .llcrypt unregistered [ 3457.744404] Key type ._llcrypt unregistered [ 3489.752670] Key type ._llcrypt registered [ 3489.755261] Key type .llcrypt registered [ 3489.909706] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3491.640263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3491.703307] alg: No test for adler32 (adler32-zlib) [ 3492.793591] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3492.989098] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3494.639221] Key type lgssc registered [ 3495.965718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3504.452741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3504.487611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.180409] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3506.201643] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.56/24@tcp [ 3506.212315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.302762] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3516.387120] LustreError: 120302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.404264] LustreError: 120302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3520.017416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.478529] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3520.546386] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.56/24@tcp [ 3520.595888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3525.614332] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3531.768156] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3535.592719] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3542.143582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3542.631695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.219719] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3551.955652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3555.374596] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3566.052121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3566.058393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.067200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3566.560196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.577183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3566.603776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3571.451666] Lustre: server umount lustre-OST0000 complete [ 3576.801602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3576.816732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3576.836591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3576.840824] Lustre: Skipped 2 previous similar messages [ 3582.287288] Lustre: server umount lustre-MDT0000 complete [ 3582.948900] LustreError: 120302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.970854] LustreError: 120302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3586.520779] LustreError: 120282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691252 with bad export cookie 11527592156248515292 [ 3586.531112] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3586.831719] Lustre: server umount lustre-MDT0001 complete [ 3596.136734] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3598.869872] Key type lgssc unregistered [ 3599.160768] LNet: 123532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3599.175031] LNetError: 123532:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3599.215686] LNet: Removed LNI 192.168.202.156@tcp [ 3600.047575] Key type .llcrypt unregistered [ 3600.048801] Key type ._llcrypt unregistered [ 3627.045846] Key type ._llcrypt registered [ 3627.047747] Key type .llcrypt registered [ 3627.149313] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3643.950288] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3644.810406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3644.914523] alg: No test for adler32 (adler32-zlib) [ 3646.124946] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3646.312113] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3647.935816] Key type lgssc registered [ 3649.416523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3686.295542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3686.338824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.822187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3687.863866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3687.946616] Lustre: lustre-MDT0000: new disk, initializing [ 3688.037930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.065025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3693.211695] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3704.200419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.361609] Lustre: 126899:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3704.421311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3704.427484] Lustre: Skipped 1 previous similar message [ 3704.490552] Lustre: lustre-MDT0001: new disk, initializing [ 3704.584576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3704.657559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3704.679788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3709.822432] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3717.854702] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3722.181663] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3729.142545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.518295] Lustre: lustre-OST0000: new disk, initializing [ 3729.524225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3729.539965] Lustre: 128643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3729.690626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.641496] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3738.187033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3738.199666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3738.333648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3745.202317] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3749.614747] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3749.800873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3754.079172] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3754.271707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3759.073383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3759.077786] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.083439] Lustre: Skipped 1 previous similar message [ 3759.086422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.228094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.237141] Lustre: Skipped 2 previous similar messages [ 3769.321886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.328871] Lustre: Skipped 1 previous similar message [ 3770.336001] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3770.546783] Lustre: server umount lustre-OST0000 complete [ 3774.432482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.436062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.441812] Lustre: Skipped 1 previous similar message [ 3779.554461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.570505] Lustre: Skipped 2 previous similar messages [ 3788.256970] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3788.436475] Lustre: server umount lustre-MDT0000 complete [ 3792.012237] LustreError: 127688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691457 with bad export cookie 3030092118145718355 [ 3792.012727] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.022913] LustreError: 127688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3792.293386] Lustre: server umount lustre-MDT0001 complete [ 3800.475633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.841508] LustreError: 130714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.856897] LustreError: 130714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3800.911056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3804.510994] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3806.177436] LustreError: 130715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.277535] LustreError: 130714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.274028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3811.483147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3814.900854] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3819.273506] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3822.533890] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3828.147978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.469974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.083311] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3837.479472] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3841.512760] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3846.975734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3850.184456] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3850.397756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3856.880347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3856.996227] Lustre: lustre-OST0001: new disk, initializing [ 3857.000120] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3857.009219] Lustre: 133746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3857.086750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3861.991608] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 3864.709194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3864.718434] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3864.844910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3868.744963] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3868.950353] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3872.499937] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3872.713197] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3875.593678] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90fe58f62000.ost_server_uuid 50 [ 3876.899274] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90fe58f62000.ost_server_uuid in FULL state after 0 sec [ 3885.537110] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.539721] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.550603] LustreError: Skipped 1 previous similar message [ 3885.565045] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3885.568843] Lustre: Skipped 3 previous similar messages [ 3889.324788] Lustre: server umount lustre-OST0001 complete [ 3890.660696] LustreError: 132310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.672756] LustreError: 132310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3893.216118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3893.220428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.226773] Lustre: Skipped 1 previous similar message [ 3895.785171] LustreError: 132309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3895.799462] LustreError: 132309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3898.650306] Lustre: server umount lustre-OST0000 complete [ 3903.968783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3903.978061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3903.997801] Lustre: Skipped 1 previous similar message [ 3904.001744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.006434] Lustre: Skipped 3 previous similar messages [ 3907.503612] Lustre: server umount lustre-MDT0000 complete [ 3910.381730] LustreError: 130695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691576 with bad export cookie 3030092118145719230 [ 3910.387537] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.395867] LustreError: 130695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3910.703892] Lustre: server umount lustre-MDT0001 complete [ 3924.836356] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 3926.992604] Key type lgssc unregistered [ 3927.252507] LNet: 136583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3927.259086] LNetError: 136583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3927.281762] LNet: Removed LNI 192.168.202.156@tcp [ 3927.909694] Key type .llcrypt unregistered [ 3927.910791] Key type ._llcrypt unregistered [ 3957.567889] Key type ._llcrypt registered [ 3957.569896] Key type .llcrypt registered [ 3957.667622] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 3967.936485] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 3968.810936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3968.843260] alg: No test for adler32 (adler32-zlib) [ 3969.802873] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3969.965028] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 3971.607171] Key type lgssc registered [ 3972.333783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3994.972309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3995.011374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.193259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3996.226198] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3996.281093] Lustre: lustre-MDT0000: new disk, initializing [ 3996.322638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3996.334142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3998.721547] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4005.915700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.988145] Lustre: 140680:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4006.032586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4006.037111] Lustre: Skipped 1 previous similar message [ 4006.093615] Lustre: lustre-MDT0001: new disk, initializing [ 4006.162830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.198823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4006.216430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4010.017582] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4015.483321] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4018.244291] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4022.748889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4022.925860] Lustre: lustre-OST0000: new disk, initializing [ 4022.929576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4022.935505] Lustre: 142419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4023.003179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.500809] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4030.526417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4030.533387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4030.616678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4033.224256] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4037.707794] Lustre: Failing over lustre-OST0000 [ 4037.829776] Lustre: server umount lustre-OST0000 complete [ 4054.224033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4054.402894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4054.410421] Lustre: Skipped 1 previous similar message [ 4054.426445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.441672] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4056.278357] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4056.480344] Lustre: 138908:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691706/real 1779691706] req@00000000dad8e565 x1866141935891456/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1779691722 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4056.499764] Lustre: 138908:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4057.503112] Lustre: 138909:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691707/real 1779691707] req@0000000099448665 x1866141935892480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779691723 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4057.521913] Lustre: 138909:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4058.980333] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4059.674582] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4059.677238] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4061.729607] Lustre: 138909:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691711/real 1779691711] req@00000000a75ad840 x1866141935893632/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779691727 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4061.759320] Lustre: 138909:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4061.917728] Lustre: Failing over lustre-MDT0000 [ 4062.155370] Lustre: server umount lustre-MDT0000 complete [ 4062.695394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4062.699395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.738381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.751741] Lustre: Skipped 1 previous similar message [ 4066.015169] Lustre: 138909:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779691716/real 1779691716] req@000000006d028cfc x1866141935894656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779691732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4069.861537] LustreError: 140687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.877278] LustreError: 140687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4071.619360] LustreError: 140688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.977290] LustreError: 140693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.994967] LustreError: 140693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4078.932646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.009838] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4079.020584] Lustre: 140685:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4079.232527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.284476] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4081.765560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4081.844157] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4084.709806] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4084.715532] Lustre: Skipped 1 previous similar message [ 4084.727789] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4084.771701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4089.826434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.830784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.840308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.850795] Lustre: Skipped 1 previous similar message [ 4089.858123] Lustre: Skipped 1 previous similar message [ 4093.655557] Lustre: server umount lustre-OST0000 complete [ 4100.072262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.074793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4100.085496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.094472] Lustre: Skipped 1 previous similar message [ 4102.503803] Lustre: server umount lustre-MDT0000 complete [ 4104.886166] LustreError: 142418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779691770 with bad export cookie 11409858032766589899 [ 4104.886954] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4104.896552] LustreError: 142418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4105.116400] Lustre: server umount lustre-MDT0001 complete [ 4112.429506] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4114.790706] Key type lgssc unregistered [ 4115.028881] LNet: 145905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4115.035739] LNetError: 145905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4115.043298] LNet: Removed LNI 192.168.202.156@tcp [ 4115.859696] Key type .llcrypt unregistered [ 4115.861584] Key type ._llcrypt unregistered [ 4144.665946] Key type ._llcrypt registered [ 4144.668111] Key type .llcrypt registered [ 4144.763847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4145.376692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4145.392827] alg: No test for adler32 (adler32-zlib) [ 4146.485574] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4146.595226] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4148.215172] Key type lgssc registered [ 4149.065869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4154.280547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4154.289081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.595732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.117733] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4163.142302] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4165.093197] LustreError: 147411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.103122] LustreError: 147411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4166.047245] LustreError: 147442:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4166.051321] LustreError: 147442:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4166.054499] LustreError: 147442:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4166.248823] Lustre: server umount lustre-MDT0000 complete [ 4175.556528] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4181.434190] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4204.846415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.018332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4205.045374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4205.107555] Lustre: lustre-MDT0000: new disk, initializing [ 4205.158313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.171572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4207.722135] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4214.934825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.032746] Lustre: 150593:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4215.056706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4215.058853] Lustre: Skipped 1 previous similar message [ 4215.112864] Lustre: lustre-MDT0001: new disk, initializing [ 4215.194067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.225068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4215.238490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4217.921101] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4223.412258] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4225.862858] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4230.191212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4230.356916] Lustre: lustre-OST0000: new disk, initializing [ 4230.362800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4230.372840] Lustre: 152332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4230.448736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4234.209747] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4238.878436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4238.882638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4238.977684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4239.405507] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4242.387439] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4243.573899] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:51:47 (1779691907) [ 4387.807681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4387.812190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4387.817141] Lustre: Skipped 1 previous similar message [ 4387.825892] Lustre: Skipped 2 previous similar messages [ 4391.009552] Lustre: server umount lustre-MDT0000 complete [ 4392.887553] LustreError: 153997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692058 with bad export cookie 13513910523845964756 [ 4392.887856] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.894280] LustreError: 153997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4392.927895] LustreError: 150603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.931192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4392.947312] Lustre: Skipped 1 previous similar message [ 4392.950216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4398.047815] LustreError: 150602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.050090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4402.150596] LustreError: 150602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.151587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4407.263373] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4407.265657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4407.265657] LustreError: 153986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.373758] Lustre: server umount lustre-MDT0001 complete [ 4415.609593] Lustre: server umount lustre-OST0000 complete [ 4420.143109] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4425.233452] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4445.092063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.233149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4445.264267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4445.311633] Lustre: lustre-MDT0000: new disk, initializing [ 4445.368993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4445.379705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4447.926663] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4454.303179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.386881] Lustre: 156825:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4454.429706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4454.433190] Lustre: Skipped 1 previous similar message [ 4454.500494] Lustre: lustre-MDT0001: new disk, initializing [ 4454.555625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4454.600074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4454.611106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4457.279207] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4461.574412] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4463.589728] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4466.995873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.103835] Lustre: lustre-OST0000: new disk, initializing [ 4467.106111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4467.109708] Lustre: 158563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4469.625870] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4473.667254] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4475.886625] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4476.959589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4476.967958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4477.015898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4478.071710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4480.192979] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4480.298815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4482.019671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.020713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4482.030047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4482.032096] LustreError: Skipped 1 previous similar message [ 4482.039511] Lustre: Skipped 1 previous similar message [ 4483.417191] Lustre: server umount lustre-OST0000 complete [ 4485.601360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4485.613132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.625696] Lustre: Skipped 1 previous similar message [ 4491.715099] Lustre: server umount lustre-MDT0000 complete [ 4492.259773] LustreError: 156837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.270731] LustreError: 156837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4493.821955] LustreError: 156818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692159 with bad export cookie 13513910523846123593 [ 4493.827438] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4493.828877] LustreError: 156818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4494.019864] Lustre: server umount lustre-MDT0001 complete [ 4498.371689] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:56:02 (1779692162) [ 4505.784407] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4510.496567] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4527.740907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.877891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4527.892448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4527.931301] Lustre: lustre-MDT0000: new disk, initializing [ 4527.977571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.983649] Lustre: Skipped 1 previous similar message [ 4527.991486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4529.947317] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4537.476739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.564302] Lustre: 163150:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4537.717477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4543.266392] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4554.683118] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4561.748945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4576.150789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4576.586232] Lustre: lustre-OST0000: new disk, initializing [ 4576.589871] Lustre: Skipped 1 previous similar message [ 4576.594791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4576.599491] Lustre: Skipped 2 previous similar messages [ 4576.606216] Lustre: 164903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4576.698668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.707147] Lustre: Skipped 1 previous similar message [ 4585.692892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4585.726582] Lustre: Skipped 1 previous similar message [ 4585.743739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4586.081816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4586.593977] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4598.675024] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4611.555320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.557192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4611.569082] Lustre: Skipped 1 previous similar message [ 4611.572168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.580277] LustreError: Skipped 1 previous similar message [ 4611.582581] Lustre: Skipped 4 previous similar messages [ 4614.940942] Lustre: server umount lustre-OST0000 complete [ 4621.792918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.832442] Lustre: Skipped 2 previous similar messages [ 4626.595423] Lustre: server umount lustre-MDT0000 complete [ 4626.917243] LustreError: 163162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.951428] LustreError: 163162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4630.575626] LustreError: 163188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692296 with bad export cookie 13513910523846124475 [ 4630.579248] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4630.581409] LustreError: 163188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4630.805147] Lustre: server umount lustre-MDT0001 complete [ 4639.189181] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:58:23 (1779692303) [ 4653.949881] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 4663.390564] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4698.540145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.773753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4698.793561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4698.887981] Lustre: lustre-MDT0000: new disk, initializing [ 4698.992996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.014963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4703.601633] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4713.368468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.508484] Lustre: 169080:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4713.751439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4718.280243] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4725.293911] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4729.175359] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.925142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4736.271311] Lustre: 170829:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4742.127787] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4746.280918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4746.459802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4749.314576] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4756.960166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4756.964700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.975315] Lustre: Skipped 1 previous similar message [ 4756.987562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4756.995746] Lustre: Skipped 2 previous similar messages [ 4761.526886] Lustre: server umount lustre-OST0000 complete [ 4771.635656] Lustre: server umount lustre-MDT0000 complete [ 4772.321676] LustreError: 169090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.344535] LustreError: 169090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4775.008669] LustreError: 169070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692440 with bad export cookie 13513910523846125560 [ 4775.009110] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.018225] LustreError: 169070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4775.480863] Lustre: server umount lustre-MDT0001 complete [ 4784.328139] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4786.766953] Key type lgssc unregistered [ 4787.006191] LNet: 172556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4787.010685] LNetError: 172556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4787.022542] LNet: Removed LNI 192.168.202.156@tcp [ 4787.852543] Key type .llcrypt unregistered [ 4787.853877] Key type ._llcrypt unregistered [ 4810.113717] Key type ._llcrypt registered [ 4810.116226] Key type .llcrypt registered [ 4810.230194] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4811.209147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4811.279823] alg: No test for adler32 (adler32-zlib) [ 4812.342405] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4812.486725] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4814.103224] Key type lgssc registered [ 4815.219406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4822.136779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4822.164607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.586620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.524853] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4833.772538] LustreError: 173966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4833.795103] LustreError: 173966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4835.909432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4836.302713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4841.177486] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4846.672024] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4850.943536] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4857.192975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4857.608605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.415376] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4868.774562] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4868.942694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4887.523524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4887.536250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.560307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.569455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4889.574958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4889.586340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.638800] Lustre: server umount lustre-OST0000 complete [ 4897.766152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4897.774714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.783821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4897.787337] Lustre: Skipped 1 previous similar message [ 4899.811676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.819654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4899.857851] Lustre: Skipped 1 previous similar message [ 4904.928383] LustreError: 173962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.948229] LustreError: 173962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4905.569775] Lustre: server umount lustre-MDT0000 complete [ 4909.840367] LustreError: 173946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692575 with bad export cookie 17468875044825882980 [ 4909.843640] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.860324] LustreError: 173946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4910.305577] Lustre: server umount lustre-MDT0001 complete [ 4920.964572] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 4923.782944] Key type lgssc unregistered [ 4924.081173] LNet: 177163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4924.088809] LNetError: 177163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4925.158963] LNet: Removed LNI 192.168.202.156@tcp [ 4925.994670] Key type .llcrypt unregistered [ 4925.996708] Key type ._llcrypt unregistered [ 4951.702276] Key type ._llcrypt registered [ 4951.706617] Key type .llcrypt registered [ 4951.841343] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 4953.090737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4953.284607] alg: No test for adler32 (adler32-zlib) [ 4954.326940] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 4954.575248] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 4956.223188] Key type lgssc registered [ 4957.325904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.696628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.724656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.190644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4970.295078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4976.612939] LustreError: 178568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.633471] LustreError: 178568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4978.818775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.188781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4983.080318] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 4988.631893] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4992.402683] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4998.555658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4998.969948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.981764] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5010.463545] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5011.248144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5018.374991] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 5019.634512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5019.641860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.653518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.664759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5021.668054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.714242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.723288] Lustre: Skipped 1 previous similar message [ 5026.800881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5026.808890] Lustre: Skipped 2 previous similar messages [ 5031.391451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5031.670718] Lustre: server umount lustre-OST0000 complete [ 5037.710434] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90fe467d4000.ost_server_uuid 50 [ 5039.364439] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5055.969364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5055.980496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.005379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5056.011184] Lustre: Skipped 1 previous similar message [ 5059.131386] Lustre: server umount lustre-MDT0000 complete [ 5062.625371] LustreError: 178563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.654481] LustreError: 178563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5062.885463] LustreError: 178548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692728 with bad export cookie 358008799490332055 [ 5062.898153] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5062.899872] LustreError: 178548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5063.305404] Lustre: server umount lustre-MDT0001 complete [ 5071.789776] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:05:35 (1779692735) [ 5078.839296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5079.319023] LustreError: 182050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5079.405766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.269706] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5084.644551] LustreError: 182051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.763452] LustreError: 182050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5094.008095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.359515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5099.551691] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5105.519971] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5109.499933] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5116.081533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5116.407816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.068852] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5124.599329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5127.420750] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.150630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.416748] Lustre: lustre-OST0001: new disk, initializing [ 5134.420556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5134.426975] Lustre: 184550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5134.548928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.597548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5140.612859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5140.766144] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5140.971266] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5149.098378] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5155.039631] Lustre: lustre-OST0000: Not available for connect from 192.168.202.56@tcp (stopping) [ 5155.047393] Lustre: Skipped 4 previous similar messages [ 5155.304159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5155.304442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.312220] LustreError: Skipped 1 previous similar message [ 5155.336941] Lustre: Skipped 2 previous similar messages [ 5158.003222] Lustre: server umount lustre-OST0000 complete [ 5160.131715] LustreError: 184559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.428676] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5163.666468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5167.301716] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5167.478742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5176.801801] LustreError: 183645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.827197] LustreError: 183645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5181.920981] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5181.925486] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.938173] LustreError: Skipped 1 previous similar message [ 5181.942465] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5181.958242] Lustre: Skipped 1 previous similar message [ 5181.977674] Lustre: Skipped 5 previous similar messages [ 5187.519994] Lustre: server umount lustre-OST0001 complete [ 5191.648423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5191.663155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.603940] Lustre: server umount lustre-MDT0000 complete [ 5200.515494] LustreError: 184548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779692866 with bad export cookie 358008799490333126 [ 5200.519648] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5200.528130] LustreError: 184548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5200.821652] Lustre: server umount lustre-MDT0001 complete [ 5221.128893] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:08:04 (1779692884) [ 5226.966178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.186489] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5227.220321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5227.367126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5230.901507] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5238.305375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.419812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5238.453888] Lustre: 187845:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5242.417649] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5250.250147] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5256.812345] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5266.815755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.148509] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5276.043262] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5276.417796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5283.711650] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5292.246557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5292.399431] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5292.712653] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5292.732553] Lustre: Skipped 2 previous similar messages [ 5300.165495] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5302.500868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5307.501604] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5312.480375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5312.484483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.501241] LustreError: Skipped 1 previous similar message [ 5312.516161] Lustre: Skipped 1 previous similar message [ 5312.517618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5312.545359] Lustre: Skipped 8 previous similar messages [ 5316.293173] Lustre: server umount lustre-OST0000 complete [ 5318.123107] LustreError: 189470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5318.144684] LustreError: 189470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5338.607297] Lustre: server umount lustre-OST0001 complete [ 5346.280624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5346.305018] LustreError: Skipped 2 previous similar messages [ 5346.306908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.333261] Lustre: Skipped 3 previous similar messages [ 5350.945162] Lustre: server umount lustre-MDT0000 complete [ 5355.932221] LustreError: 187836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693021 with bad export cookie 358008799490334533 [ 5355.940376] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5355.949418] LustreError: 187836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5356.563873] Lustre: server umount lustre-MDT0001 complete [ 5381.850300] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:10:45 (1779693045) [ 5400.577682] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 5411.568837] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 5452.778343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.000684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5453.016740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5453.101718] Lustre: lustre-MDT0000: new disk, initializing [ 5453.194875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.222660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5458.219085] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5469.945578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.140251] Lustre: 195453:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5470.199593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5470.204554] Lustre: Skipped 1 previous similar message [ 5470.296461] Lustre: lustre-MDT0001: new disk, initializing [ 5470.435426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5470.453872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5475.737799] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5483.748742] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5487.998815] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5496.047260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.451797] Lustre: lustre-OST0000: new disk, initializing [ 5496.455099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5496.463190] Lustre: 197196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5503.044380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5503.059056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5503.210136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5504.461498] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5513.145279] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5519.182812] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5519.564119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5524.883555] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5525.101609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5529.057265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5529.058255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.071658] LustreError: Skipped 1 previous similar message [ 5529.072704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.072708] Lustre: Skipped 6 previous similar messages [ 5529.138263] Lustre: Skipped 2 previous similar messages [ 5541.343537] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5541.583122] Lustre: server umount lustre-OST0000 complete [ 5552.714316] Lustre: server umount lustre-MDT0000 complete [ 5554.656815] LustreError: 197150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5554.672170] LustreError: 197150:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5556.734932] LustreError: 195446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693222 with bad export cookie 358008799490336318 [ 5556.759432] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5557.284422] Lustre: server umount lustre-MDT0001 complete [ 5567.579923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.853420] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5583.822676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.230039] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5596.879159] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5601.277911] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5608.722334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5616.912921] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5623.637509] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5630.084453] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5630.331911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5636.002261] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5636.320492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5645.351766] Lustre: server umount lustre-OST0000 complete [ 5652.407990] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5658.805992] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5690.144948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5693.754943] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.778883] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5696.868569] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5702.915438] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5706.199563] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5708.133969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.143476] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5713.376593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5713.383901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.384470] LustreError: Skipped 3 previous similar messages [ 5713.390282] Lustre: Skipped 4 previous similar messages [ 5713.401594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.412528] Lustre: Skipped 12 previous similar messages [ 5716.305607] Lustre: server umount lustre-OST0000 complete [ 5726.948642] Lustre: server umount lustre-MDT0000 complete [ 5728.748295] LustreError: 201974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5728.781076] LustreError: 201974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5732.382532] LustreError: 199247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693398 with bad export cookie 358008799490337200 [ 5732.395979] LustreError: 199247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5732.405709] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5732.824873] Lustre: server umount lustre-MDT0001 complete [ 5743.325602] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:16:46 (1779693406) [ 5752.091236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5752.802526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5752.809380] Lustre: Skipped 6 previous similar messages [ 5758.430505] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5768.418767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.228220] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5780.108779] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5784.036680] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5791.527455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5799.134328] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5805.191371] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5810.304488] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5810.573635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5815.712945] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5815.983716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5823.002246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.223384] Lustre: lustre-OST0001: new disk, initializing [ 5823.227376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5823.233089] Lustre: 207710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5829.399426] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5830.824230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5830.843276] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5830.949988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5838.126830] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5843.932920] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5844.384456] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5848.980573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5849.245817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5857.748740] Lustre: server umount lustre-OST0001 complete [ 5865.021685] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5865.276899] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5871.336226] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5871.715807] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5901.227933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.931072] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5906.948653] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5906.952893] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5909.877573] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 5917.293384] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5919.490932] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5927.912738] Lustre: server umount lustre-OST0001 complete [ 5951.619334] Lustre: server umount lustre-OST0000 complete [ 5961.825426] Lustre: server umount lustre-MDT0000 complete [ 5966.118519] LustreError: 204697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693632 with bad export cookie 358008799490338166 [ 5966.121026] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5966.132657] LustreError: 204697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5972.587170] Lustre: server umount lustre-MDT0001 complete [ 6000.936535] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:21:04 (1779693664) [ 6009.054313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.494706] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6009.552637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6015.132936] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6024.987643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.134549] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6025.164604] Lustre: 212668:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6030.800975] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6037.947075] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6043.048842] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6050.893053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6051.170601] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6057.605466] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6063.216297] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6072.195993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6072.248118] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6079.400309] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6085.829603] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6087.912984] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6093.183869] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6093.418213] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6098.281277] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6098.436717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6102.090530] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90fe445e1800.ost_server_uuid 50 [ 6103.618739] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90fe445e1800.ost_server_uuid in IDLE state after 0 sec [ 6104.500529] Lustre: Permanently deactivating lustre-OST0001 [ 6104.509583] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6107.902710] Lustre: Permanently reactivating lustre-OST0001 [ 6114.278645] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6114.279426] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.293415] LustreError: Skipped 7 previous similar messages [ 6114.294303] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6114.294309] Lustre: Skipped 16 previous similar messages [ 6114.346616] Lustre: Skipped 12 previous similar messages [ 6118.628590] Lustre: server umount lustre-OST0001 complete [ 6119.399930] LustreError: 214288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6119.417577] LustreError: 214288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 6136.287136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6136.400371] Lustre: server umount lustre-OST0000 complete [ 6146.452879] Lustre: server umount lustre-MDT0000 complete [ 6150.190826] LustreError: 212661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693816 with bad export cookie 358008799490339454 [ 6150.192819] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6150.202704] LustreError: 212661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6150.603338] Lustre: server umount lustre-MDT0001 complete [ 6178.711884] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:24:02 (1779693842) [ 6186.995399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.533463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6187.560606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6187.572618] Lustre: Skipped 5 previous similar messages [ 6192.583075] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6201.940314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.025572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6202.060699] Lustre: 218776:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6207.254485] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6213.019901] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6216.302311] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6223.821759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6224.019056] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6224.044993] Lustre: Permanently deactivating lustre-OST0000 [ 6224.054714] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6230.420457] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6232.098698] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6237.033903] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6244.816829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6244.904904] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6251.697348] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6255.361371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6257.719373] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6259.666980] Lustre: Permanently reactivating lustre-OST0000 [ 6259.682990] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6259.688979] Lustre: Skipped 2 previous similar messages [ 6262.256616] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6262.341983] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6262.348877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.366968] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6262.374948] Lustre: Skipped 1 previous similar message [ 6262.393552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6287.327464] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6287.572592] Lustre: server umount lustre-OST0001 complete [ 6298.639465] Lustre: server umount lustre-OST0000 complete [ 6309.912461] Lustre: server umount lustre-MDT0000 complete [ 6314.490072] LustreError: 218766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779693980 with bad export cookie 358008799490341099 [ 6314.498226] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6314.503512] LustreError: 218766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6314.793300] Lustre: server umount lustre-MDT0001 complete [ 6326.535677] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6329.899994] Key type lgssc unregistered [ 6330.265307] LNet: 223186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6330.275673] LNetError: 223186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6331.327159] LNet: Removed LNI 192.168.202.156@tcp [ 6332.259240] Key type .llcrypt unregistered [ 6332.260858] Key type ._llcrypt unregistered [ 6360.157721] Key type ._llcrypt registered [ 6360.160400] Key type .llcrypt registered [ 6360.282530] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6361.817985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6361.846059] alg: No test for adler32 (adler32-zlib) [ 6362.982667] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 6363.232615] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6364.927258] Key type lgssc registered [ 6366.437034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6376.821666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6376.838688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.259515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.461671] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6388.706663] LustreError: 224638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.725829] LustreError: 224638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6393.829151] LustreError: 224639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.098343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.318620] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6394.367903] Lustre: 224627:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6394.391245] Lustre: Permanently deactivating lustre-MDT0001 [ 6394.423107] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6395.004568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6400.020099] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6400.537600] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6406.113317] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6410.773691] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6419.043584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6419.434156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.466750] LustreError: 226255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.476042] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6423.529481] LustreError: 226258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6423.550740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6423.556682] LustreError: 226258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6426.386307] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6428.648648] LustreError: 226258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.685210] LustreError: 226258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6432.279349] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6438.893486] LustreError: 226313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.907790] LustreError: 226313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6439.458506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6439.848326] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6445.070560] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6445.083181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6446.789736] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6454.728624] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6458.254204] Lustre: Permanently reactivating lustre-MDT0001 [ 6458.277515] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6458.283176] Lustre: Skipped 2 previous similar messages [ 6461.544746] Lustre: lustre-MDT0001: Client 18ff9a38-3e90-45ce-84f8-504884d6a4eb (at 192.168.202.56@tcp) reconnecting [ 6465.790855] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.811498] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6465.823898] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6465.832489] LustreError: 227819:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6465.836920] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6465.842637] LustreError: 227819:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6465.848857] LustreError: 227819:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.156@tcp: cfg command failed: rc = -108 [ 6465.856685] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6465.856685] [ 6465.865366] LustreError: 224631:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6466.000176] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6468.314947] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6475.640360] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6475.974912] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6482.218990] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6482.522803] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6485.738591] Lustre: Permanently deactivating lustre-MDT0001 [ 6485.756424] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6485.761540] Lustre: Skipped 2 previous similar messages [ 6492.328294] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6501.346493] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6501.358056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6501.370055] Lustre: Skipped 3 previous similar messages [ 6501.380034] Lustre: Skipped 3 previous similar messages [ 6501.619581] Lustre: server umount lustre-MDT0000 complete [ 6505.660757] LustreError: 225362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694171 with bad export cookie 15205250287373480227 [ 6505.678094] LustreError: 225362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6505.678779] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6505.952829] Lustre: server umount lustre-MDT0001 complete [ 6512.297840] Lustre: server umount lustre-OST0000 complete [ 6524.302714] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:29:47 (1779694187) [ 6536.019747] Lustre: server umount lustre-OST0001 complete [ 6542.923338] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 6554.053789] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6590.890511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.074852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6591.080309] Lustre: Skipped 2 previous similar messages [ 6591.100531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6591.173475] Lustre: lustre-MDT0000: new disk, initializing [ 6591.237033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.252131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6595.203352] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6605.742906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.890898] Lustre: 232281:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6605.942420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6605.945989] Lustre: Skipped 1 previous similar message [ 6606.011739] Lustre: lustre-MDT0001: new disk, initializing [ 6606.131747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6606.169536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6606.183195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6611.414727] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6618.826045] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6623.137447] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6629.851390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6630.146173] Lustre: lustre-OST0000: new disk, initializing [ 6630.148849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6630.154807] Lustre: 234024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6630.250460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.210640] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6639.143389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6639.149675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6639.242646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6643.871884] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6646.354462] LustreError: 232290:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6656.324755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6656.391113] LustreError: 232290:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6656.406833] LustreError: 232291:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6656.609362] Lustre: lustre-OST0001: new disk, initializing [ 6656.614173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6656.625881] Lustre: 235025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6663.198626] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6666.396600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6666.409506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6666.439754] LustreError: 232291:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6666.522179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6671.305270] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6673.224473] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6676.697993] Lustre: lustre-OST0001: Not available for connect from 192.168.202.56@tcp (stopping) [ 6676.963295] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.969331] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.630977] Lustre: server umount lustre-OST0001 complete [ 6681.791242] LustreError: 234007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6700.514506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6700.518616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.530963] LustreError: Skipped 2 previous similar messages [ 6700.548208] Lustre: Skipped 1 previous similar message [ 6700.553521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6700.553530] Lustre: Skipped 2 previous similar messages [ 6702.568581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6702.583300] Lustre: Skipped 2 previous similar messages [ 6704.504304] Lustre: server umount lustre-OST0000 complete [ 6708.705447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6708.718131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.734960] Lustre: Skipped 1 previous similar message [ 6708.747747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6715.112451] Lustre: server umount lustre-MDT0000 complete [ 6717.921256] LustreError: 232295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.947044] LustreError: 232295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6718.877361] LustreError: 232274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694384 with bad export cookie 15205250287373483041 [ 6718.880793] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.891332] LustreError: 232274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6733.279303] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6733.630236] Lustre: server umount lustre-MDT0001 complete [ 6744.822539] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 6747.854755] Key type lgssc unregistered [ 6748.274884] LNet: 236926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6748.283847] LNetError: 236926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6748.319228] LNet: Removed LNI 192.168.202.156@tcp [ 6749.539572] Key type .llcrypt unregistered [ 6749.542422] Key type ._llcrypt unregistered [ 6786.178420] Key type ._llcrypt registered [ 6786.180462] Key type .llcrypt registered [ 6786.278551] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 6787.627451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6787.648884] alg: No test for adler32 (adler32-zlib) [ 6788.656553] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 6788.831145] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 6790.472343] Key type lgssc registered [ 6791.712861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6799.227811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6799.243947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.489631] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6800.511189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6800.784800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.997783] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6811.796824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.921801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6811.973675] Lustre: 239098:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6812.245894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6816.274990] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6820.993198] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6824.965640] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6833.449506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.576235] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6833.778364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6838.922881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6839.740072] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6845.299819] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6900.206192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6900.216376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.230659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6900.727483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6903.039902] Lustre: server umount lustre-OST0000 complete [ 6907.638848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6918.767120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6919.131820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.193521] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6920.211544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6920.214425] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6925.651718] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 6931.453986] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6974.663437] Lustre: server umount lustre-OST0000 complete [ 6981.090625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6981.097400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.108550] Lustre: Skipped 1 previous similar message [ 6981.114282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.120215] Lustre: Skipped 1 previous similar message [ 6986.726281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6986.732102] Lustre: Skipped 2 previous similar messages [ 6989.026278] Lustre: server umount lustre-MDT0000 complete [ 6992.791286] LustreError: 239090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694658 with bad export cookie 5413390964141694967 [ 6992.810078] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7002.227401] Lustre: server umount lustre-MDT0001 complete [ 7011.351269] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7013.936724] Key type lgssc unregistered [ 7014.174675] LNet: 244050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7014.187897] LNetError: 244050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7015.270631] LNet: Removed LNI 192.168.202.156@tcp [ 7015.975986] Key type .llcrypt unregistered [ 7015.978161] Key type ._llcrypt unregistered [ 7039.539013] Key type ._llcrypt registered [ 7039.543215] Key type .llcrypt registered [ 7039.638724] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7040.670498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7040.719599] alg: No test for adler32 (adler32-zlib) [ 7041.871528] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 7042.115252] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7043.801828] Key type lgssc registered [ 7044.859570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7051.252856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7051.272532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.629630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7056.649077] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7063.014731] LustreError: 245453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.026441] LustreError: 245453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7063.880971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7064.276277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7068.196426] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7074.338845] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7078.212290] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7084.546525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7084.885103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.669292] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7095.750661] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7098.236087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7123.939177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.953170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7123.966358] Lustre: Skipped 1 previous similar message [ 7123.979858] Lustre: Skipped 1 previous similar message [ 7128.981930] Lustre: server umount lustre-OST0000 complete [ 7134.176460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.202104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.211029] Lustre: Skipped 1 previous similar message [ 7138.804822] Lustre: server umount lustre-MDT0000 complete [ 7139.298463] LustreError: 245447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.311520] LustreError: 245447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7142.250215] LustreError: 245432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694808 with bad export cookie 2860838484699102844 [ 7142.263035] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7142.263652] LustreError: 245432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7142.545177] Lustre: server umount lustre-MDT0001 complete [ 7152.363389] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7154.613848] Key type lgssc unregistered [ 7154.897080] LNet: 249400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7154.907667] LNetError: 249400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7155.952741] LNet: Removed LNI 192.168.202.156@tcp [ 7156.856871] Key type .llcrypt unregistered [ 7156.859302] Key type ._llcrypt unregistered [ 7173.111921] Key type ._llcrypt registered [ 7173.114383] Key type .llcrypt registered [ 7173.194602] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7174.250937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7174.294734] alg: No test for adler32 (adler32-zlib) [ 7175.363669] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 7175.557529] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7177.223254] Key type lgssc registered [ 7178.229763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7185.849322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7185.866076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.466143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7192.272750] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7197.680542] LustreError: 250606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7197.721073] LustreError: 250606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7201.233322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.579540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7205.596309] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7211.256732] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7215.499388] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7221.792167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.095726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7227.711368] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7232.310454] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7233.387237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7246.816353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7246.821361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.834815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7248.870500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7248.872645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7248.873668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.896442] Lustre: Skipped 1 previous similar message [ 7252.921916] Lustre: server umount lustre-OST0000 complete [ 7258.080687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7258.089974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7258.102419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7262.714449] Lustre: server umount lustre-MDT0000 complete [ 7264.231636] LustreError: 251627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7264.255416] LustreError: 251627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7266.322634] LustreError: 250588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779694932 with bad export cookie 620324385109012243 [ 7266.324814] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7266.333155] LustreError: 250588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7266.792331] Lustre: server umount lustre-MDT0001 complete [ 7276.673343] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7278.852358] Key type lgssc unregistered [ 7279.050980] LNet: 253927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7279.062621] LNetError: 253927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7279.072846] LNet: Removed LNI 192.168.202.156@tcp [ 7279.795505] Key type .llcrypt unregistered [ 7279.798086] Key type ._llcrypt unregistered [ 7302.365823] Key type ._llcrypt registered [ 7302.370882] Key type .llcrypt registered [ 7302.506583] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7303.463994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7303.511967] alg: No test for adler32 (adler32-zlib) [ 7304.558817] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 7304.806702] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7306.463202] Key type lgssc registered [ 7307.585173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7315.072526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7315.096820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7316.535939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.891847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7326.700274] LustreError: 255319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.734685] LustreError: 255319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7329.845232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.159584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7334.389456] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7339.253431] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7343.249547] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7348.966885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7349.221309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.624835] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7358.041740] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7361.516220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7386.080529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7386.087990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7386.106512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7387.107542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7387.114249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.115910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7390.160502] Lustre: server umount lustre-OST0000 complete [ 7396.832247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7396.837773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.846225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7396.849417] Lustre: Skipped 1 previous similar message [ 7402.466473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7402.473477] Lustre: Skipped 3 previous similar messages [ 7407.585076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7407.594308] Lustre: Skipped 1 previous similar message [ 7408.095233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7408.384772] Lustre: server umount lustre-MDT0000 complete [ 7412.455134] LustreError: 255299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695078 with bad export cookie 13014493170624226190 [ 7412.458753] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7412.465868] LustreError: 255299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7412.884786] Lustre: server umount lustre-MDT0001 complete [ 7423.205678] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7425.580947] Key type lgssc unregistered [ 7425.854644] LNet: 259270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7425.859504] LNetError: 259270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7425.877759] LNet: Removed LNI 192.168.202.156@tcp [ 7426.659956] Key type .llcrypt unregistered [ 7426.665207] Key type ._llcrypt unregistered [ 7442.928560] Key type ._llcrypt registered [ 7442.930589] Key type .llcrypt registered [ 7443.047688] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7443.958074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7443.992257] alg: No test for adler32 (adler32-zlib) [ 7445.186347] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 7445.470633] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7447.167533] Key type lgssc registered [ 7448.323539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7454.747323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7454.760333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.195488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.416497] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7466.466110] LustreError: 260504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7466.485761] LustreError: 260504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7468.265216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.691233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7472.394479] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7477.636777] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7481.147614] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7488.865315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7489.146584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7495.884157] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7500.554796] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7501.433880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7517.164538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7517.165656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.170131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7517.189023] Lustre: Skipped 1 previous similar message [ 7518.452457] Lustre: server umount lustre-OST0000 complete [ 7524.842164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7524.853896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.876670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.880166] Lustre: Skipped 1 previous similar message [ 7527.396639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.398573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7527.428229] Lustre: Skipped 1 previous similar message [ 7528.966095] Lustre: server umount lustre-MDT0000 complete [ 7532.464473] LustreError: 260469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695198 with bad export cookie 14921464267495548338 [ 7532.471847] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7532.481292] LustreError: 260469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7532.996407] Lustre: server umount lustre-MDT0001 complete [ 7545.570559] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7548.089428] Key type lgssc unregistered [ 7548.361953] LNet: 263847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7548.376418] LNetError: 263847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7548.391769] LNet: Removed LNI 192.168.202.156@tcp [ 7549.416811] Key type .llcrypt unregistered [ 7549.420755] Key type ._llcrypt unregistered [ 7560.976322] Key type ._llcrypt registered [ 7560.978425] Key type .llcrypt registered [ 7561.091965] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7577.047248] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 7593.241833] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7595.594745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7595.743975] alg: No test for adler32 (adler32-zlib) [ 7596.832651] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 7597.080726] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7598.807173] Key type lgssc registered [ 7600.283168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7636.169458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7636.215137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.501048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7637.523395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7637.604501] Lustre: lustre-MDT0000: new disk, initializing [ 7637.703895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7637.722889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7642.897306] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7654.098199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7654.256315] Lustre: 267351:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7654.310787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7654.316088] Lustre: Skipped 1 previous similar message [ 7654.426584] Lustre: lustre-MDT0001: new disk, initializing [ 7654.527383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7654.561744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7654.586721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7659.643809] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7667.290933] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7671.065945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7677.634240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7677.888678] Lustre: lustre-OST0000: new disk, initializing [ 7677.892887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7677.898169] Lustre: 269092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7677.977380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7684.583971] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7687.779368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7687.806265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7687.905826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7692.136316] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7696.707513] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7696.887673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7700.857571] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7701.118251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7703.521236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7703.523272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.529258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.531532] LustreError: Skipped 1 previous similar message [ 7703.582219] Lustre: Skipped 1 previous similar message [ 7708.641780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7708.647718] Lustre: Skipped 2 previous similar messages [ 7709.653121] Lustre: server umount lustre-OST0000 complete [ 7713.763292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7713.788910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7713.798878] Lustre: Skipped 1 previous similar message [ 7718.881385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.889170] Lustre: Skipped 1 previous similar message [ 7719.415392] Lustre: server umount lustre-MDT0000 complete [ 7723.051311] LustreError: 267341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695389 with bad export cookie 13025969793313875617 [ 7723.051311] LustreError: 267340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695389 with bad export cookie 13025969793313875617 [ 7723.057818] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.060161] LustreError: 267340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7723.412925] Lustre: server umount lustre-MDT0001 complete [ 7731.078072] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:49:55 (1779695395) [ 7736.639616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.227306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7741.713916] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7742.433909] LustreError: 271358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.454538] LustreError: 271358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7747.554504] LustreError: 271357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7750.453678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.663410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7755.137916] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7761.791563] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7765.996871] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7773.349659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.528779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.338935] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7785.304955] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7796.192360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7796.201072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.209952] Lustre: Skipped 1 previous similar message [ 7796.213546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7798.243782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7798.246946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.666377] Lustre: server umount lustre-OST0000 complete [ 7806.948469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7806.961988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7806.981193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7806.991313] Lustre: Skipped 2 previous similar messages [ 7810.526421] Lustre: server umount lustre-MDT0000 complete [ 7813.600384] LustreError: 271354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.608502] LustreError: 271354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7815.247486] LustreError: 271339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695481 with bad export cookie 13025969793313876499 [ 7815.251392] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.262700] LustreError: 271339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7815.809469] Lustre: server umount lustre-MDT0001 complete [ 7828.045469] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 7830.866671] Key type lgssc unregistered [ 7831.121420] LNet: 274537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7831.128826] LNetError: 274537:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7831.149386] LNet: Removed LNI 192.168.202.156@tcp [ 7831.856366] Key type .llcrypt unregistered [ 7831.857703] Key type ._llcrypt unregistered [ 7857.571156] Key type ._llcrypt registered [ 7857.573315] Key type .llcrypt registered [ 7857.704675] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 7873.094846] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 7874.413923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7874.488611] alg: No test for adler32 (adler32-zlib) [ 7875.663358] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 7875.811415] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 7877.423634] Key type lgssc registered [ 7879.151787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7920.827638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7920.851299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7922.024654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7922.044742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7922.082270] Lustre: lustre-MDT0000: new disk, initializing [ 7922.120926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.127604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7925.338582] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7939.893236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.015235] Lustre: 278331:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7940.099335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7940.104618] Lustre: Skipped 1 previous similar message [ 7940.216742] Lustre: lustre-MDT0001: new disk, initializing [ 7940.399788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7940.471738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7940.497720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7945.581118] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7952.823717] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7956.634822] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7963.721135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.005838] Lustre: lustre-OST2710: new disk, initializing [ 7964.010243] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7964.024214] Lustre: 280076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7964.157921] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7969.857666] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7972.933794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7972.948239] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7973.135811] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7977.707660] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7984.080375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7984.224462] Lustre: lustre-OST03e8: new disk, initializing [ 7984.227738] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7984.237451] Lustre: 281027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7984.327043] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7990.575849] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 7992.441936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7992.457566] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7992.640558] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7996.830868] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8004.685219] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8004.895159] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8009.722971] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8009.938459] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8013.553261] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8013.692411] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8018.465355] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8018.740536] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8026.067715] Lustre: server umount lustre-MDT0000 complete [ 8027.622378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8027.628177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.646533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.659461] Lustre: Skipped 1 previous similar message [ 8031.046378] LustreError: 278324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695697 with bad export cookie 9838958893777714289 [ 8031.068497] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8031.660338] Lustre: server umount lustre-MDT0001 complete [ 8045.177798] Lustre: server umount lustre-OST2710 complete [ 8058.950317] Lustre: server umount lustre-OST03e8 complete [ 8078.858575] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 8088.875297] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8123.561837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.716210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8123.761855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8123.826345] Lustre: lustre-MDT0000: new disk, initializing [ 8123.914806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8123.929668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8128.145393] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8140.704900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.864899] Lustre: 286060:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8140.949570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8140.953467] Lustre: Skipped 1 previous similar message [ 8141.082506] Lustre: lustre-MDT0001: new disk, initializing [ 8141.236978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8141.330946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8141.349866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8146.586278] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8155.821967] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8160.385787] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8167.733258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8168.089116] Lustre: lustre-OST0000: new disk, initializing [ 8168.100651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8168.111947] Lustre: 287803:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8168.231985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.719875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8174.728871] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8174.850149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8175.523393] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8184.036743] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8189.288666] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8189.632270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8195.652423] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8195.898338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8200.672346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8200.677608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.681776] LustreError: Skipped 1 previous similar message [ 8200.683444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8200.714823] Lustre: Skipped 2 previous similar messages [ 8204.394882] Lustre: server umount lustre-OST0000 complete [ 8210.915279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.936304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8210.940062] Lustre: Skipped 2 previous similar messages [ 8214.595168] Lustre: server umount lustre-MDT0000 complete [ 8216.034362] LustreError: 286072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.084774] LustreError: 286072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8220.101792] LustreError: 289198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695886 with bad export cookie 9838958893777715962 [ 8220.115721] LustreError: 289198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8220.117228] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8220.580763] Lustre: server umount lustre-MDT0001 complete [ 8229.072857] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:58:12 (1779695892) [ 8230.350423] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8231.788730] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:58:15 (1779695895) [ 8235.430712] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8265.617092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8265.971176] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8266.010474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8266.171762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.326883] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8281.193394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.305604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8281.379158] Lustre: 291499:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8281.567314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8287.918717] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8312.949535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8313.493582] Lustre: Denying initial registration attempt from nid 192.168.202.156@tcp, specified as failover [ 8313.518880] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8313.533270] LustreError: 292919:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8313.540422] LustreError: 292919:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8313.559497] LustreError: 292919:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8313.737466] Lustre: server umount lustre-OST0000 complete [ 8313.749329] LustreError: 292919:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8323.045446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8323.069094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8323.069532] Lustre: Skipped 2 previous similar messages [ 8323.070087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8323.114055] Lustre: Skipped 1 previous similar message [ 8328.160990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8328.172061] Lustre: Skipped 1 previous similar message [ 8329.071481] Lustre: server umount lustre-MDT0000 complete [ 8332.554620] LustreError: 291490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779695998 with bad export cookie 9838958893777716851 [ 8332.557832] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8332.562666] LustreError: 291490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8332.873375] Lustre: server umount lustre-MDT0001 complete [ 8345.595570] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8348.580556] Key type lgssc unregistered [ 8348.898749] LNet: 294025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8348.911905] LNetError: 294025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8348.936739] LNet: Removed LNI 192.168.202.156@tcp [ 8349.898795] Key type .llcrypt unregistered [ 8349.901073] Key type ._llcrypt unregistered [ 8361.593864] Key type ._llcrypt registered [ 8361.595204] Key type .llcrypt registered [ 8361.737299] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8362.682547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8362.758080] alg: No test for adler32 (adler32-zlib) [ 8363.920287] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 8364.332324] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8366.079164] Key type lgssc registered [ 8367.836963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8419.740119] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8430.105619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8430.148050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.676580] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8431.721329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8431.934158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.045756] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8446.001210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8446.127582] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8446.178748] Lustre: 296777:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8446.460845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8451.453409] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8458.518392] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8462.805400] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8482.808516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8482.950987] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8483.171507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8489.430262] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8495.852301] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8500.705637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8500.710508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.744941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8501.220385] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8501.229277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8501.248466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8501.255319] Lustre: Skipped 1 previous similar message [ 8504.688287] Lustre: server umount lustre-OST0000 complete [ 8511.459140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.463205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.492505] Lustre: Skipped 1 previous similar message [ 8511.507031] Lustre: Skipped 1 previous similar message [ 8516.576588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.593641] Lustre: Skipped 1 previous similar message [ 8521.696942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.709395] Lustre: Skipped 1 previous similar message [ 8523.231578] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8523.459647] Lustre: server umount lustre-MDT0000 complete [ 8526.880848] LustreError: 296768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696192 with bad export cookie 9613950231552044965 [ 8526.891257] LustreError: 296768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8526.895123] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8527.184217] Lustre: server umount lustre-MDT0001 complete [ 8537.929286] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8541.134532] Key type lgssc unregistered [ 8541.553764] LNet: 299993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8541.558551] LNetError: 299993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8541.593347] LNet: Removed LNI 192.168.202.156@tcp [ 8542.696132] Key type .llcrypt unregistered [ 8542.700181] Key type ._llcrypt unregistered [ 8568.250840] Key type ._llcrypt registered [ 8568.252639] Key type .llcrypt registered [ 8568.393946] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8569.904830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8570.174935] alg: No test for adler32 (adler32-zlib) [ 8571.302953] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 8571.593097] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8573.415197] Key type lgssc registered [ 8574.924611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8584.075374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8584.088783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.497459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.132757] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8595.939139] LustreError: 301395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.959464] LustreError: 301395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8599.229333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.739487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.391317] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8611.812238] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8615.833773] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8624.645222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8625.119403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8631.801980] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8637.597953] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8654.817651] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.829301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8657.436953] Lustre: server umount lustre-MDT0000 complete [ 8659.939615] LustreError: 301395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8659.948215] LustreError: 301395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8661.431350] LustreError: 301376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696327 with bad export cookie 2530749118470882606 [ 8661.449615] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8665.056905] LustreError: 301391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8665.061219] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8665.065129] LustreError: 301391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8665.083566] Lustre: Skipped 2 previous similar messages [ 8665.086869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8665.090705] Lustre: Skipped 3 previous similar messages [ 8667.808966] Lustre: server umount lustre-MDT0001 complete [ 8674.072546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8684.044588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.239476] LustreError: 304655:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8686.247673] LustreError: 304655:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000103b5384 x1866146761449984/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1779696352 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8686.280783] LustreError: 304655:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 8686.816342] LustreError: 302998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.830938] LustreError: 302998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8686.927635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.075645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8691.823658] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8692.013386] LustreError: 302999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8692.042030] LustreError: 302999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8696.098395] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8700.038991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8700.483725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8705.511396] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8705.627641] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8711.381357] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8713.138704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8716.906989] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8718.566666] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8722.913652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8722.916517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.922403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8722.925961] Lustre: Skipped 2 previous similar messages [ 8725.988216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8725.992186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8726.012635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8731.112147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8731.121155] Lustre: Skipped 2 previous similar messages [ 8737.248467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8737.438253] Lustre: server umount lustre-OST0000 complete [ 8746.478102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8746.504357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8746.504570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8746.516302] Lustre: Skipped 1 previous similar message [ 8746.542378] Lustre: Skipped 3 previous similar messages [ 8756.191326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8756.604967] Lustre: server umount lustre-MDT0000 complete [ 8756.706902] LustreError: 304668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.727482] LustreError: 304668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8761.275382] LustreError: 303860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696427 with bad export cookie 2530749118470890271 [ 8761.283430] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8761.285671] LustreError: 303860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8761.681719] Lustre: server umount lustre-MDT0001 complete [ 8775.104292] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 8778.932612] Key type lgssc unregistered [ 8779.283232] LNet: 307103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8779.298048] LNetError: 307103:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8779.330103] LNet: Removed LNI 192.168.202.156@tcp [ 8780.275527] Key type .llcrypt unregistered [ 8780.277251] Key type ._llcrypt unregistered [ 8803.600851] Key type ._llcrypt registered [ 8803.602328] Key type .llcrypt registered [ 8803.782466] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 8804.782596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8804.845480] alg: No test for adler32 (adler32-zlib) [ 8806.144356] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 8806.418535] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 8808.066478] Key type lgssc registered [ 8809.321510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8818.274415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8818.296214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.962656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.879398] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8829.411249] LustreError: 308502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8829.435780] LustreError: 308502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8834.471811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8834.529074] LustreError: 308501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8834.959521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8840.903078] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8847.085669] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8851.405812] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8854.503407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8854.510677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8854.515131] Lustre: Skipped 1 previous similar message [ 8859.617084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8859.619540] Lustre: Skipped 1 previous similar message [ 8860.643957] LustreError: 308496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.670908] LustreError: 308496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8860.794175] Lustre: server umount lustre-MDT0000 complete [ 8864.357593] LustreError: 308481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696530 with bad export cookie 5074719708154463024 [ 8864.365896] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8864.368035] LustreError: 308481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8864.748834] Lustre: server umount lustre-MDT0001 complete [ 8880.167496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8880.616698] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8880.690742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8880.997436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8885.767275] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8895.019946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8895.167601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8895.197346] Lustre: 310865:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8895.523686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8900.568330] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8907.277356] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8912.148145] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8922.864810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.133359] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8923.147872] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8923.156754] LustreError: 312522:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 8923.161772] LustreError: 312522:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8923.165829] LustreError: 312522:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8923.289588] Lustre: server umount lustre-OST0000 complete [ 8923.293108] LustreError: 312522:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8934.381753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8934.637782] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8934.840765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.911126] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8943.859821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8948.144292] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8957.081488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8957.236104] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8957.316995] Lustre: lustre-OST0001: new disk, initializing [ 8957.319507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8957.323070] Lustre: 313951:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8957.446492] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8963.183166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8963.198622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8963.361399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8964.335742] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 8969.470387] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8973.797853] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8973.805357] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8973.824875] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8978.347644] Lustre: server umount lustre-OST0001 complete [ 8978.929799] LustreError: 312990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.973176] LustreError: 312990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8984.037576] LustreError: 312988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8984.039894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8984.045559] LustreError: 312988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8984.055929] Lustre: Skipped 1 previous similar message [ 8984.059714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8984.062324] Lustre: Skipped 1 previous similar message [ 8989.067191] Lustre: server umount lustre-OST0000 complete [ 8994.272927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8994.301195] Lustre: Skipped 1 previous similar message [ 8994.313154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8994.323271] Lustre: Skipped 1 previous similar message [ 9000.947674] Lustre: server umount lustre-MDT0000 complete [ 9004.513159] LustreError: 310870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9004.522170] LustreError: 310870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9005.827520] LustreError: 310857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696671 with bad export cookie 5074719708154463577 [ 9005.841311] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9005.850290] LustreError: 310857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9006.550645] Lustre: server umount lustre-MDT0001 complete [ 9019.613057] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9022.830492] Key type lgssc unregistered [ 9023.072777] LNet: 315779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9023.082833] LNetError: 315779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9024.102212] LNet: Removed LNI 192.168.202.156@tcp [ 9025.351397] Key type .llcrypt unregistered [ 9025.354977] Key type ._llcrypt unregistered [ 9098.344680] Key type ._llcrypt registered [ 9098.349749] Key type .llcrypt registered [ 9098.542310] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9111.302924] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9112.196633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9112.222215] alg: No test for adler32 (adler32-zlib) [ 9113.337644] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 9113.676079] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9115.423423] Key type lgssc registered [ 9116.901969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9151.579129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9151.615314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.873292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9152.899345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9152.988173] Lustre: lustre-MDT0000: new disk, initializing [ 9153.074428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9153.098489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9156.720773] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9166.645425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.820814] Lustre: 320684:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9166.870106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9166.873417] Lustre: Skipped 1 previous similar message [ 9166.930298] Lustre: lustre-MDT0001: new disk, initializing [ 9167.018926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9167.053931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9167.069344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9171.221886] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9179.415393] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9183.511511] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9191.255785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.552331] Lustre: lustre-OST0000: new disk, initializing [ 9191.557184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9191.565110] Lustre: 322423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9191.732511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9197.990050] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9201.726754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9201.733813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9201.820688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9204.828289] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9208.676635] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9208.852636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9214.095143] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9214.303451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9217.506578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9217.525903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.531488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9217.576669] Lustre: Skipped 1 previous similar message [ 9222.651137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9222.687782] Lustre: Skipped 2 previous similar messages [ 9223.073710] Lustre: server umount lustre-OST0000 complete [ 9227.745289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9227.746508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9227.771754] Lustre: Skipped 1 previous similar message [ 9227.789989] Lustre: Skipped 1 previous similar message [ 9232.866681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9232.878218] Lustre: Skipped 1 previous similar message [ 9233.935451] Lustre: server umount lustre-MDT0000 complete [ 9237.651650] LustreError: 320676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779696903 with bad export cookie 10355612108662176368 [ 9237.655143] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9237.661225] LustreError: 320676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9238.070267] Lustre: server umount lustre-MDT0001 complete [ 9246.232045] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:15:09 (1779696909) [ 9253.405664] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:15:17 (1779696917) [ 9262.617756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.077242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.844800] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9268.210846] LustreError: 325030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9268.227304] LustreError: 325030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9273.312455] LustreError: 325031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9273.656937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9274.042354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9278.130393] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9283.156644] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9286.812452] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9293.095709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9293.363754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9298.576885] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9303.925945] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9315.514097] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9316.983208] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9318.409273] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9319.912649] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9321.615709] Lustre: Failing over lustre-MDT0000 [ 9321.926459] Lustre: server umount lustre-MDT0000 complete [ 9324.002535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9324.007572] LustreError: 325026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9324.020055] Lustre: Skipped 1 previous similar message [ 9324.042279] LustreError: 325026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9329.121021] LustreError: 325031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9329.140779] LustreError: 325031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9334.249997] LustreError: 325749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9334.265501] LustreError: 325749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9339.360230] Lustre: 318904:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779696989/real 1779696989] req@00000000b1584e52 x1866147329802112/t0(0) o400->MGC192.168.202.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779697005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9339.378575] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9340.505786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9344.481795] LustreError: 325749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.495872] LustreError: 325749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 9349.603122] LustreError: 318903:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e27806cd x1866147329807232/t0(0) o250->MGC192.168.202.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9349.990907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9350.059525] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9351.871152] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9354.844801] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9355.260046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9355.280864] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9355.372844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9362.701495] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9364.522858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9368.157536] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9370.076283] Lustre: Failing over lustre-MDT0000 [ 9370.436504] Lustre: server umount lustre-MDT0000 complete [ 9370.596063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9370.598200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9370.602243] LustreError: Skipped 1 previous similar message [ 9370.602951] LustreError: 325749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.602958] LustreError: 325749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9370.680644] Lustre: Skipped 3 previous similar messages [ 9388.617419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.712455] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9388.772632] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (not set up) [ 9388.990044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.048822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9392.672115] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9393.862858] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9394.161894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9394.168100] Lustre: Skipped 2 previous similar messages [ 9394.190896] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9394.262089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9400.328402] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9402.063341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9409.507525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.515297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9409.527132] Lustre: Skipped 2 previous similar messages [ 9409.555914] Lustre: Skipped 2 previous similar messages [ 9413.888476] Lustre: server umount lustre-MDT0000 complete [ 9414.625788] LustreError: 325030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.638939] LustreError: 325030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9417.177616] LustreError: 325012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697083 with bad export cookie 10355612108662179084 [ 9417.201363] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9417.606565] Lustre: server umount lustre-MDT0001 complete [ 9430.382727] Lustre: server umount lustre-OST0000 complete [ 9444.768319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.446941] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9457.171419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.426421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9457.436669] Lustre: Skipped 1 previous similar message [ 9461.242450] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9465.863641] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9469.160432] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9474.879232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.942480] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9485.845543] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9490.615353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9496.035130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.037504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.038907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.042442] Lustre: Skipped 1 previous similar message [ 9500.728329] Lustre: server umount lustre-OST0000 complete [ 9511.979738] Lustre: server umount lustre-MDT0000 complete [ 9516.421574] LustreError: 330804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697182 with bad export cookie 10355612108662179833 [ 9516.430547] LustreError: 330804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9516.438469] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.766878] Lustre: server umount lustre-MDT0001 complete [ 9528.496085] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9531.146583] Key type lgssc unregistered [ 9531.443569] LNet: 333999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9531.448425] LNetError: 333999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9531.464858] LNet: Removed LNI 192.168.202.156@tcp [ 9532.540115] Key type .llcrypt unregistered [ 9532.542051] Key type ._llcrypt unregistered [ 9560.543403] Key type ._llcrypt registered [ 9560.546713] Key type .llcrypt registered [ 9560.742825] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9562.005498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9562.023610] alg: No test for adler32 (adler32-zlib) [ 9562.946561] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 9563.188714] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9564.831939] Key type lgssc registered [ 9565.802350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9573.440964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9573.454986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.859591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.578649] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9584.109362] LustreError: 335501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9584.141100] LustreError: 335501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9587.461047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9587.899681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9592.396946] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9597.558404] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9601.315340] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9608.525932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9608.951179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.366642] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9619.201195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9620.437222] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9633.264915] Lustre: server umount lustre-MDT0000 complete [ 9634.271953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9634.278540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.294145] LustreError: 335501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.785448] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9636.909116] LustreError: 335483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697302 with bad export cookie 10171946754741928291 [ 9636.910795] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9636.919480] LustreError: 335483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9637.245954] Lustre: server umount lustre-MDT0001 complete [ 9651.074153] Lustre: server umount lustre-OST0000 complete [ 9662.873892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.204899] LustreError: 338869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9663.228921] LustreError: 338869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9663.285529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.734720] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9668.582882] LustreError: 338870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9673.700255] LustreError: 338869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9674.658983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.023525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.163264] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9686.102037] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9690.693530] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9697.658443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.073878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.560347] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9708.343750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9708.863594] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9723.878687] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9723.885238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9723.912924] Lustre: Skipped 2 previous similar messages [ 9723.951688] Lustre: Skipped 2 previous similar messages [ 9728.829513] Lustre: server umount lustre-MDT0000 complete [ 9728.994285] LustreError: 340077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9729.036216] LustreError: 340077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9733.777959] LustreError: 341123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697399 with bad export cookie 10171946754741929404 [ 9733.785231] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9733.793058] LustreError: 341123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9734.164991] Lustre: server umount lustre-MDT0001 complete [ 9747.275956] Lustre: server umount lustre-OST0000 complete [ 9762.684896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.241372] LustreError: 342440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.321272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9768.179965] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9776.546804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.992563] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9788.286536] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9791.368967] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9798.309686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9798.631190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.635086] Lustre: Skipped 1 previous similar message [ 9807.360135] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [ 9808.931646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9813.110398] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9822.177625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9822.180685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.190770] Lustre: Skipped 1 previous similar message [ 9822.199526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.226704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9824.241465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9824.252947] Lustre: Skipped 1 previous similar message [ 9826.158360] Lustre: server umount lustre-OST0000 complete [ 9829.345703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.347338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9829.354877] Lustre: Skipped 2 previous similar messages [ 9829.382710] Lustre: Skipped 1 previous similar message [ 9834.466058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9834.479312] Lustre: Skipped 1 previous similar message [ 9835.302548] Lustre: server umount lustre-MDT0000 complete [ 9839.589990] LustreError: 342437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.603124] LustreError: 342437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9839.793810] LustreError: 342421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697505 with bad export cookie 10171946754741930426 [ 9839.804627] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9839.805525] LustreError: 342421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9840.172676] Lustre: server umount lustre-MDT0001 complete [ 9853.063509] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9856.374953] Key type lgssc unregistered [ 9856.699359] LNet: 345618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9856.721333] LNetError: 345618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9856.742624] LNet: Removed LNI 192.168.202.156@tcp [ 9857.793435] Key type .llcrypt unregistered [ 9857.795575] Key type ._llcrypt unregistered [ 9882.706917] Key type ._llcrypt registered [ 9882.709225] Key type .llcrypt registered [ 9882.830211] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9884.292222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9884.328815] alg: No test for adler32 (adler32-zlib) [ 9885.425636] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 9885.648119] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9887.343169] Key type lgssc registered [ 9888.693318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9897.777136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9897.863029] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9897.889864] LustreError: 347046:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9897.941202] LustreError: 347046:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9897.949704] LustreError: 347046:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9897.965257] LustreError: 347046:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9897.988814] LustreError: 347046:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9906.781315] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9906.787963] LustreError: 347443:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9906.802218] LustreError: 347443:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9906.806814] LustreError: 347443:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9906.811918] LustreError: 347443:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9906.816926] LustreError: 347443:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9928.551649] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [ 9931.299384] Key type lgssc unregistered [ 9931.504814] LNet: 348430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9931.507600] LNetError: 348430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9931.513832] LNet: Removed LNI 192.168.202.156@tcp [ 9932.067537] Key type .llcrypt unregistered [ 9932.069971] Key type ._llcrypt unregistered [ 9948.009823] Key type ._llcrypt registered [ 9948.016622] Key type .llcrypt registered [ 9948.126240] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [ 9962.209754] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [ 9963.373170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9963.396904] alg: No test for adler32 (adler32-zlib) [ 9964.537242] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 9964.741557] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [ 9966.479774] Key type lgssc registered [ 9967.484759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9999.088930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9999.112306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10000.549984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10000.615362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10000.712302] Lustre: lustre-MDT0000: new disk, initializing [10000.890933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10000.923988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10005.161512] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10019.054832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10019.215626] Lustre: 351551:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10019.326103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10019.339249] Lustre: Skipped 1 previous similar message [10019.452321] Lustre: lustre-MDT0001: new disk, initializing [10019.638178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10019.751168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10019.759853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10024.352713] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10031.495794] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10034.800099] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10041.650397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10041.891860] Lustre: lustre-OST0000: new disk, initializing [10041.896174] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10041.900954] Lustre: 353289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10042.038782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10047.067524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10047.075378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10047.205065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10047.921756] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10055.231750] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10059.550668] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10059.739184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10064.756149] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10064.995363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10067.935914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10067.947359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.959217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10067.959874] Lustre: Skipped 1 previous similar message [10067.986935] Lustre: Skipped 1 previous similar message [10073.061983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10073.066594] Lustre: Skipped 1 previous similar message [10078.195688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10078.205930] Lustre: Skipped 1 previous similar message [10081.249257] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10081.376343] Lustre: server umount lustre-OST0000 complete [10086.372409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10086.375471] LustreError: Skipped 1 previous similar message [10086.383103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10086.396874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10088.418058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.571555] Lustre: server umount lustre-MDT0000 complete [10093.540747] LustreError: 351562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10093.560897] LustreError: 351562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10095.788425] LustreError: 351541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697761 with bad export cookie 541085312436806362 [10095.790581] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10095.816847] LustreError: 351541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10096.230716] Lustre: server umount lustre-MDT0001 complete [10105.412167] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10107.768224] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:29:30 (1779697770) [10114.409941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10114.838431] LustreError: 355604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.915858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10120.051363] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10120.175413] LustreError: 355605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10124.271337] LustreError: 355604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10129.377854] LustreError: 355605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.932797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10131.287240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10137.936505] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10143.330161] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10147.040707] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10153.572375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10153.907133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10160.480489] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10167.125723] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10174.558090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10174.769672] Lustre: lustre-OST0001: new disk, initializing [10174.780392] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10174.806163] Lustre: 358102:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10174.921301] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10181.862242] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10184.921158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10184.930868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10185.032417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10190.592365] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10195.430584] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10195.435812] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10195.458332] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10195.477687] Lustre: Skipped 1 previous similar message [10195.502434] Lustre: Skipped 3 previous similar messages [10201.088168] Lustre: server umount lustre-OST0001 complete [10202.823407] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.026704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.030156] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.070365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10221.080969] LustreError: 357200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10221.092166] Lustre: Skipped 3 previous similar messages [10233.311115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10233.561865] Lustre: server umount lustre-OST0000 complete [10238.944561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10238.960726] LustreError: Skipped 1 previous similar message [10238.965805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10238.993273] Lustre: Skipped 1 previous similar message [10239.000467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.009384] Lustre: Skipped 5 previous similar messages [10244.303224] Lustre: server umount lustre-MDT0000 complete [10248.888391] LustreError: 358101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779697914 with bad export cookie 541085312436807244 [10248.902627] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10248.904620] LustreError: 358101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10255.294517] Lustre: server umount lustre-MDT0001 complete [10267.329379] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10270.241478] Key type lgssc unregistered [10270.490653] LNet: 359960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10270.503363] LNetError: 359960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10270.529495] LNet: Removed LNI 192.168.202.156@tcp [10271.229147] Key type .llcrypt unregistered [10271.231124] Key type ._llcrypt unregistered [10332.042034] Key type ._llcrypt registered [10332.044481] Key type .llcrypt registered [10332.254800] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10334.055953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10334.157702] alg: No test for adler32 (adler32-zlib) [10335.438869] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [10335.817499] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10337.487968] Key type lgssc registered [10338.786037] Lustre: Echo OBD driver; http://www.lustre.org/ [10347.420902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10347.449548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.717419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10348.738661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10348.977909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10353.682836] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10364.281967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.407212] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10364.479384] Lustre: 362754:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10364.957069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10370.312879] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10376.647665] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10380.861368] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10385.376615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10385.381183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.400715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10386.401962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10386.405812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10389.425387] Lustre: server umount lustre-MDT0000 complete [10393.881038] LustreError: 362745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698059 with bad export cookie 196862849336834826 [10393.921414] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10394.349939] Lustre: server umount lustre-MDT0001 complete [10406.450736] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:34:29 (1779698069) [10414.395631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.140852] LustreError: 365040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.161242] LustreError: 365040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10415.251226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10420.720165] LustreError: 365041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.348111] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10425.830715] LustreError: 365040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.945835] LustreError: 365041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.377693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.901940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10437.943218] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10443.734425] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10448.218769] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10455.358857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.576296] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10455.751807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10462.225983] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10468.233291] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10475.801585] Lustre: Permanently deactivating lustre-OST0000 [10475.823736] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10485.799895] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10487.728950] LustreError: 367736:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10487.739236] LustreError: 367736:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10487.743374] LustreError: 367736:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10490.885550] LustreError: 367785:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10490.889746] LustreError: 367785:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10498.777477] Lustre: server umount lustre-OST0000 complete [10501.801911] LustreError: 368034:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [10501.806301] LustreError: 368034:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10504.673336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10504.686233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10504.709742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.715389] Lustre: Skipped 1 previous similar message [10508.775748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.782046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.889421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.895122] Lustre: Skipped 2 previous similar messages [10516.996983] Lustre: server umount lustre-MDT0000 complete [10519.016203] LustreError: 366989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.040611] LustreError: 366989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10520.797603] LustreError: 365020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698186 with bad export cookie 196862849336835407 [10520.799029] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10520.808571] LustreError: 365020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10530.453450] Lustre: server umount lustre-MDT0001 complete [10539.497753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.453829] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10547.530095] LustreError: 369177:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10555.592914] Lustre: 369516:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.156@tcp [10558.017752] Lustre: server umount MGS complete [10571.721868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.361682] LustreError: 370240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10572.398183] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10572.402540] Lustre: Skipped 1 previous similar message [10572.561270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10577.285847] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10586.260331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.476530] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10591.536512] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10597.947789] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10602.398373] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10604.435983] Lustre: Permanently reactivating lustre-OST0000 [10604.460886] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10604.468516] Lustre: Skipped 2 previous similar messages [10633.184876] LustreError: 371796:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10642.073333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10642.413485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10642.421097] Lustre: Skipped 1 previous similar message [10649.228231] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10655.152976] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10662.883288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10662.895391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.935276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10675.168110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10675.333193] Lustre: server umount lustre-OST0000 complete [10683.873461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10683.888980] LustreError: Skipped 1 previous similar message [10683.893642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.909016] Lustre: Skipped 1 previous similar message [10683.915200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.923764] Lustre: Skipped 6 previous similar messages [10685.858739] Lustre: server umount lustre-MDT0000 complete [10689.505193] LustreError: 370241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.528650] LustreError: 370241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10690.198168] LustreError: 370222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698356 with bad export cookie 196862849336836457 [10690.219379] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10690.219939] LustreError: 370222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10690.479281] Lustre: server umount lustre-MDT0001 complete [10702.220886] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [10705.620776] Key type lgssc unregistered [10705.945246] LNet: 374249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10705.951291] LNetError: 374249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10705.967317] LNet: Removed LNI 192.168.202.156@tcp [10706.934879] Key type .llcrypt unregistered [10706.936790] Key type ._llcrypt unregistered [10728.160593] Key type ._llcrypt registered [10728.166582] Key type .llcrypt registered [10728.316304] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [10747.151433] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [10749.126057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10749.230434] alg: No test for adler32 (adler32-zlib) [10750.820670] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [10751.220479] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [10753.007342] Key type lgssc registered [10754.170453] Lustre: Echo OBD driver; http://www.lustre.org/ [10792.170377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10792.201826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10793.538932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10793.592135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10793.685443] Lustre: lustre-MDT0000: new disk, initializing [10793.837700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10793.857726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10798.344655] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10811.409295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10811.581169] Lustre: 377373:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10811.646797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10811.651252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10811.652634] Lustre: Skipped 1 previous similar message [10811.750571] Lustre: lustre-MDT0001: new disk, initializing [10811.866764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10811.926468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10811.942731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10817.203427] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10826.063893] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10830.672085] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10840.073117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10840.538747] Lustre: lustre-OST0000: new disk, initializing [10840.543937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10840.561576] Lustre: 379116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10840.788983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10848.387773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10848.406156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10848.610650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10848.814856] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10858.463927] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10862.421651] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10862.546740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10867.828811] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10868.080522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10874.338231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10874.342639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10874.347286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10874.365967] LustreError: Skipped 1 previous similar message [10874.384455] Lustre: Skipped 1 previous similar message [10876.055400] Lustre: server umount lustre-OST0000 complete [10883.561351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10883.566526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10883.582365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10883.587239] Lustre: Skipped 1 previous similar message [10884.579668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.113482] Lustre: server umount lustre-MDT0000 complete [10891.367994] LustreError: 377365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698557 with bad export cookie 8898921080684067170 [10891.379184] LustreError: 377365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10891.379490] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10891.899288] Lustre: server umount lustre-MDT0001 complete [10901.480434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10903.840848] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:42:47 (1779698567) [10912.670269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.477374] LustreError: 381431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.510349] LustreError: 381431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10913.634870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10918.698687] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10918.884154] LustreError: 381432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.005210] LustreError: 381431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.458964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.920215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10933.661157] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10939.430595] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10943.054636] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10950.772060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.173971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10958.505719] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [10965.474920] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10967.727829] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10970.595644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10977.760442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10977.772138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10977.796132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10977.803899] Lustre: Skipped 2 previous similar messages [10979.811756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10982.242558] Lustre: server umount lustre-OST0000 complete [10989.536631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10989.545505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.557518] Lustre: Skipped 1 previous similar message [10989.562518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10989.565193] Lustre: Skipped 2 previous similar messages [11000.292617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11000.303516] Lustre: Skipped 5 previous similar messages [11001.311429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11001.595865] Lustre: server umount lustre-MDT0000 complete [11004.404252] LustreError: 381426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.432842] LustreError: 381426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11006.340749] LustreError: 383046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698672 with bad export cookie 8898921080684068052 [11006.350601] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11006.753301] Lustre: server umount lustre-MDT0001 complete [11017.131290] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11020.013378] Key type lgssc unregistered [11020.439385] LNet: 384798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11020.450678] LNetError: 384798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11020.483613] LNet: Removed LNI 192.168.202.156@tcp [11021.757393] Key type .llcrypt unregistered [11021.759215] Key type ._llcrypt unregistered [11050.849535] Key type ._llcrypt registered [11050.852334] Key type .llcrypt registered [11051.074195] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11051.470447] Key type .llcrypt unregistered [11051.472109] Key type ._llcrypt unregistered [11071.962185] Key type ._llcrypt registered [11071.967270] Key type .llcrypt registered [11072.148186] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11073.505360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11073.689302] alg: No test for adler32 (adler32-zlib) [11074.909943] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [11075.234464] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11077.095150] Key type lgssc registered [11078.950314] Lustre: Echo OBD driver; http://www.lustre.org/ [11090.726297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11090.741522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.359447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11097.429559] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11102.690487] LustreError: 386979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.702133] LustreError: 386979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11107.238095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.814489] LustreError: 386980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11107.951879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11113.445363] LustreError: 386979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.459232] LustreError: 386979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11115.714848] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11117.551805] LustreError: 386980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11117.570869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11117.577349] LustreError: 386980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11121.990272] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11122.667216] LustreError: 386979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.678292] LustreError: 386979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11130.233800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11130.793984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11136.240646] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11146.211271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.219542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11146.245300] Lustre: Skipped 1 previous similar message [11146.273354] Lustre: Skipped 1 previous similar message [11147.454623] Lustre: server umount lustre-OST0000 complete [11156.462341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11156.463185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.465205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11156.505669] Lustre: Skipped 1 previous similar message [11158.283736] Lustre: server umount lustre-MDT0000 complete [11161.580652] LustreError: 386975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11161.591954] LustreError: 386975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11162.615661] LustreError: 386961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779698828 with bad export cookie 1720096137682416306 [11162.622851] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11162.631539] LustreError: 386961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11163.282831] Lustre: server umount lustre-MDT0001 complete [11174.720474] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11177.837917] Key type lgssc unregistered [11178.021981] LNet: 389961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11178.025747] LNetError: 389961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11178.041732] LNet: Removed LNI 192.168.202.156@tcp [11178.990205] Key type .llcrypt unregistered [11178.998946] Key type ._llcrypt unregistered [11209.563651] Key type ._llcrypt registered [11209.568156] Key type .llcrypt registered [11209.751814] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11211.642178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11211.893314] alg: No test for adler32 (adler32-zlib) [11213.175260] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [11213.417659] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11215.143984] Key type lgssc registered [11217.081691] Lustre: Echo OBD driver; http://www.lustre.org/ [11227.557765] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11227.616376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11243.359709] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11248.480755] LustreError: 391335:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.156@tcp: failed processing log, type 4: rc = -110 [11275.168591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11283.009101] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11290.412368] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11297.629238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.436973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11298.558880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11303.929380] LustreError: 391360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.959601] LustreError: 391360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11304.344942] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11309.035097] LustreError: 392261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11309.079305] LustreError: 392261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11314.149576] LustreError: 392259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.171562] LustreError: 392259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11314.510248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11314.978563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11321.021926] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11327.116922] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11331.506518] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11339.748885] Lustre: server umount lustre-OST0000 complete [11345.888459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11345.898234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11345.898366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11345.909743] Lustre: Skipped 1 previous similar message [11345.943695] Lustre: Skipped 1 previous similar message [11351.011857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11351.028799] Lustre: Skipped 1 previous similar message [11353.176096] Lustre: server umount lustre-MDT0000 complete [11356.127255] Lustre: 390671:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699006/real 1779699006] req@00000000c032a180 x1866149531776768/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779699022 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11356.130874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.143112] LustreError: 392265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.145751] Lustre: 390671:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11357.372720] LustreError: 391341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699023 with bad export cookie 847193845216239669 [11357.379622] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11357.384283] LustreError: 391341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11357.709705] Lustre: server umount lustre-MDT0001 complete [11370.485966] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11374.422977] Key type lgssc unregistered [11374.863766] LNet: 394548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11374.867297] LNetError: 394548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11375.908901] LNet: Removed LNI 192.168.202.156@tcp [11376.814118] Key type .llcrypt unregistered [11376.821849] Key type ._llcrypt unregistered [11407.150165] Key type ._llcrypt registered [11407.154593] Key type .llcrypt registered [11407.276457] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11408.921954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11409.067703] alg: No test for adler32 (adler32-zlib) [11410.149990] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [11410.535754] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11412.272937] Key type lgssc registered [11413.680207] Lustre: Echo OBD driver; http://www.lustre.org/ [11423.529897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11423.549345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11425.267760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11431.142190] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11435.493340] LustreError: 395953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.523482] LustreError: 395953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11440.622926] LustreError: 395954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11440.831893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.243575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11446.359499] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11452.550899] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11456.866236] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11464.897262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.243971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11470.416814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11472.033857] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11477.615612] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11482.081050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11482.089428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11482.100160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11484.384277] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.56@tcp (stopping) [11485.664809] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11485.671970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11485.685278] Lustre: Skipped 1 previous similar message [11485.703541] Lustre: Skipped 2 previous similar messages [11488.457600] Lustre: server umount lustre-MDT0000 complete [11489.470196] LustreError: 395948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.676599] LustreError: 395935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.56@tcp arrived at 1779699157 with bad export cookie 17084866369376333794 [11493.816326] LustreError: 395935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699159 with bad export cookie 17084866369376332898 [11493.823473] LustreError: 395935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11493.829399] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11495.906729] LustreError: 396665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.907215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.930677] LustreError: 396665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11495.979766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11500.087454] Lustre: server umount lustre-OST0000 complete [11504.048459] LustreError: 396665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11504.069359] LustreError: 396665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11508.121169] Lustre: server umount lustre-MDT0001 complete [11522.204472] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11525.806311] Key type lgssc unregistered [11526.168849] LNet: 399281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11526.203364] LNetError: 399281:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11526.226373] LNet: Removed LNI 192.168.202.156@tcp [11527.231274] Key type .llcrypt unregistered [11527.233931] Key type ._llcrypt unregistered [11556.638558] Key type ._llcrypt registered [11556.643321] Key type .llcrypt registered [11556.792376] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11558.300628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11558.396275] alg: No test for adler32 (adler32-zlib) [11559.609212] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [11559.889313] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11561.560077] Key type lgssc registered [11563.160731] Lustre: Echo OBD driver; http://www.lustre.org/ [11572.362986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11572.413505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.345716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11580.725883] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11584.491829] LustreError: 400683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.537742] LustreError: 400683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11589.604150] LustreError: 400684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11591.164857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.488650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11597.904943] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11603.369118] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11607.186878] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11615.638225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11615.950922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11620.019430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11622.950687] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11629.728199] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11635.691733] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11635.703981] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11637.166895] Lustre: server umount lustre-MDT0001 complete [11638.027894] LustreError: 400678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.805179] LustreError: 400683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.846708] LustreError: 400683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11645.921703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11645.924016] LustreError: 400684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.933313] Lustre: Skipped 1 previous similar message [11645.940928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11645.965495] Lustre: Skipped 1 previous similar message [11651.357556] Lustre: server umount lustre-OST0000 complete [11656.320827] Lustre: server umount lustre-MDT0000 complete [11672.453226] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11676.238797] Key type lgssc unregistered [11676.557757] LNet: 404011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11676.561311] LNetError: 404011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11677.608407] LNet: Removed LNI 192.168.202.156@tcp [11678.509444] Key type .llcrypt unregistered [11678.517082] Key type ._llcrypt unregistered [11704.150774] Key type ._llcrypt registered [11704.152141] Key type .llcrypt registered [11704.283083] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [11719.251293] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11720.461442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11720.564182] alg: No test for adler32 (adler32-zlib) [11721.631164] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [11722.004303] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [11723.671346] Key type lgssc registered [11725.408340] Lustre: Echo OBD driver; http://www.lustre.org/ [11767.275541] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [11784.990368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11785.022582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11786.207788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11786.226696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11786.280726] Lustre: lustre-MDT0000: new disk, initializing [11786.343543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11786.357459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11791.300202] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11807.522173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11807.837746] Lustre: lustre-OST0000: new disk, initializing [11807.841623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11807.845088] Lustre: Skipped 1 previous similar message [11807.853802] Lustre: 408979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11807.919765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11814.940208] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11815.541846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11815.549040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11815.587876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11823.894881] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11842.716902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11842.971817] Lustre: 407980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11843.118057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11843.270367] Lustre: lustre-MDT0001: new disk, initializing [11843.513689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11843.610848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11843.643171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11850.095091] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [11860.074645] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11860.348961] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11865.971557] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11866.241822] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11873.543351] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [11874.785467] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.788475] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11874.797911] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11874.812676] Lustre: Skipped 1 previous similar message [11878.605216] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [11883.711646] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.56@tcp (stopping) [11883.713724] Lustre: Skipped 2 previous similar messages [11886.048214] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11886.335432] Lustre: server umount lustre-MDT0001 complete [11895.266620] LustreError: 407984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.297939] LustreError: 407984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11900.395192] LustreError: 407986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.419738] LustreError: 407986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11905.511910] LustreError: 407990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11905.514455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11905.568984] Lustre: Skipped 1 previous similar message [11905.576559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11905.585779] Lustre: Skipped 2 previous similar messages [11910.638456] LustreError: 407989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.744473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11915.744678] LustreError: 407990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.750154] Lustre: Skipped 1 previous similar message [11919.327442] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11919.675343] Lustre: server umount lustre-OST0000 complete [11938.271141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11938.714333] Lustre: server umount lustre-MDT0000 complete [11953.208868] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11956.943813] Key type lgssc unregistered [11957.376052] LNet: 412559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11957.385363] LNetError: 412559:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11957.404632] LNet: Removed LNI 192.168.202.156@tcp [11958.339897] Key type .llcrypt unregistered [11958.342931] Key type ._llcrypt unregistered [11999.451436] Key type ._llcrypt registered [11999.453793] Key type .llcrypt registered [11999.569262] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [11999.840479] Key type .llcrypt unregistered [11999.842610] Key type ._llcrypt unregistered [12017.477138] Key type ._llcrypt registered [12017.478675] Key type .llcrypt registered [12017.593681] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [12018.669098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12018.693472] alg: No test for adler32 (adler32-zlib) [12019.733793] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [12019.954755] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [12021.623196] Key type lgssc registered [12022.966559] Lustre: Echo OBD driver; http://www.lustre.org/ [12046.294802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12046.331942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.530025] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12047.560775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12047.644842] Lustre: lustre-MDT0000: new disk, initializing [12047.795628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12047.815720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12052.226673] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12063.519134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.651795] Lustre: 415914:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12063.704465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12063.708438] Lustre: Skipped 1 previous similar message [12063.796221] Lustre: lustre-MDT0001: new disk, initializing [12063.929821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12063.981981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12063.997330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12070.348214] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12081.224045] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12084.562933] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12093.150497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.666346] Lustre: lustre-OST0000: new disk, initializing [12093.672328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12093.679248] Lustre: 417659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12093.856186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.163986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12100.177252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12100.266586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12101.406581] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12109.375840] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12115.428798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12115.437311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.461460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12115.944169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.966156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12121.059811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12121.063296] Lustre: Skipped 1 previous similar message [12123.445791] Lustre: server umount lustre-MDT0000 complete [12128.077697] LustreError: 416694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699794 with bad export cookie 3467278119350831724 [12128.080432] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12128.085235] LustreError: 416694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12128.337487] Lustre: server umount lustre-MDT0001 complete [12142.800450] Lustre: server umount lustre-OST0000 complete [12154.762297] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:03:38 (1779699818) [12162.861488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.308136] LustreError: 419574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12163.331562] LustreError: 419574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12163.442589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12168.518823] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12168.678711] LustreError: 419575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12173.814311] LustreError: 419574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12177.561402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12178.038534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12183.378395] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12190.038397] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12194.767272] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12215.209729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12215.563768] Lustre: Found index 0 for lustre-OST0000, updating log [12215.803054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12222.987382] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12229.805455] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12234.297854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12241.897360] Lustre: server umount lustre-OST0000 complete [12259.809110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.811498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12259.812635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12259.812640] Lustre: Skipped 2 previous similar messages [12259.822791] Lustre: Skipped 2 previous similar messages [12260.831355] Lustre: 414446:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699909/real 1779699909] req@00000000e01d2f5f x1866150376969728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779699925 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12265.952747] Lustre: 414449:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699915/real 1779699915] req@0000000098d948aa x1866150376971648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779699931 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12265.953982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12265.984692] Lustre: 414449:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12266.001425] Lustre: Skipped 4 previous similar messages [12269.120161] Lustre: server umount lustre-MDT0000 complete [12270.056023] LustreError: 419574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.088680] LustreError: 419574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12270.097086] Lustre: 414448:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779699920/real 1779699920] req@00000000b858a92d x1866150376972672/t0(0) o400->lustre-OST0000-osc-MDT0001@1.2.3.4@tcp:28/4 lens 224/224 e 0 to 1 dl 1779699936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12273.243484] LustreError: 419556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779699939 with bad export cookie 3467278119350833124 [12273.251581] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12273.251741] LustreError: 419556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12273.619098] Lustre: server umount lustre-MDT0001 complete [12282.188640] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:05:45 (1779699945) [12282.208099] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [12905.334254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12906.190225] LustreError: 425649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12906.211696] LustreError: 425649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12906.367408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12911.614300] LustreError: 425650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12913.096797] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12916.752677] LustreError: 425649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12918.829587] LustreError: 425650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12918.881495] LustreError: 425650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12923.489941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12923.877596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12924.121065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12930.867896] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12937.366645] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12941.291853] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12961.121747] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [12963.319392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12963.905340] Lustre: Found index 0 for lustre-OST0000, updating log [12964.625303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12974.129896] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [12981.640544] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12984.964766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12984.974808] Lustre: Skipped 1 previous similar message [12984.983951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12990.289842] Lustre: server umount lustre-OST0000 complete [12995.424780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12995.428777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12995.446528] Lustre: Skipped 1 previous similar message [12999.585484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12999.595593] Lustre: Skipped 8 previous similar messages [13000.611511] LustreError: 425644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.627428] LustreError: 425644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13000.763343] Lustre: server umount lustre-MDT0000 complete [13005.073602] LustreError: 425629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779700671 with bad export cookie 3467278119350834209 [13005.080467] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13005.090719] LustreError: 425629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13005.492664] Lustre: server umount lustre-MDT0001 complete [13021.538772] LNet: 1 local NIs in recovery (showing 1): 192.168.202.156@tcp [13038.908692] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [13050.744383] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13093.659743] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:19:17 (1779700757) [13114.185910] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13117.991061] Key type lgssc unregistered [13118.339676] LNet: 431871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13118.347971] LNetError: 431871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13227.942326] LNet: Removed LNI 192.168.202.156@tcp [13228.636081] Key type .llcrypt unregistered [13228.637728] Key type ._llcrypt unregistered [13245.318356] Key type ._llcrypt registered [13245.320827] Key type .llcrypt registered [13245.425384] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13246.524249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13246.805050] alg: No test for adler32 (adler32-zlib) [13247.831973] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [13248.054596] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13249.752247] Key type lgssc registered [13250.865046] Lustre: Echo OBD driver; http://www.lustre.org/ [13255.111371] LNet: Added LNI 192.168.202.156@tcp100 [8/256/0/180] [13256.009260] LNet: Added LNI 192.168.202.156@tcp101 [8/256/0/180] [13257.044163] LNet: Added LNI 192.168.202.156@tcp102 [8/256/0/180] [13257.763747] LNet: Added LNI 192.168.202.156@tcp103 [8/256/0/180] [13258.453486] LNet: Added LNI 192.168.202.156@tcp104 [8/256/0/180] [13259.207765] LNet: Added LNI 192.168.202.156@tcp105 [8/256/0/180] [13259.971977] LNet: Added LNI 192.168.202.156@tcp106 [8/256/0/180] [13260.818444] LNet: Added LNI 192.168.202.156@tcp107 [8/256/0/180] [13261.880047] LNet: Added LNI 192.168.202.156@tcp108 [8/256/0/180] [13262.944831] LNet: Added LNI 192.168.202.156@tcp109 [8/256/0/180] [13263.858149] LNet: Added LNI 192.168.202.156@tcp110 [8/256/0/180] [13264.698685] LNet: Added LNI 192.168.202.156@tcp111 [8/256/0/180] [13265.795761] LNet: Added LNI 192.168.202.156@tcp112 [8/256/0/180] [13266.703848] LNet: Added LNI 192.168.202.156@tcp113 [8/256/0/180] [13267.605606] LNet: Added LNI 192.168.202.156@tcp114 [8/256/0/180] [13268.681659] LNet: Added LNI 192.168.202.156@tcp115 [8/256/0/180] [13269.661212] LNet: Added LNI 192.168.202.156@tcp116 [8/256/0/180] [13270.711585] LNet: Added LNI 192.168.202.156@tcp117 [8/256/0/180] [13271.802716] LNet: Added LNI 192.168.202.156@tcp118 [8/256/0/180] [13272.777941] LNet: Added LNI 192.168.202.156@tcp119 [8/256/0/180] [13273.952291] LNet: Added LNI 192.168.202.156@tcp120 [8/256/0/180] [13275.302808] LNet: Added LNI 192.168.202.156@tcp121 [8/256/0/180] [13276.549478] LNet: Added LNI 192.168.202.156@tcp122 [8/256/0/180] [13277.689928] LNet: Added LNI 192.168.202.156@tcp123 [8/256/0/180] [13278.474446] LNet: Added LNI 192.168.202.156@tcp124 [8/256/0/180] [13279.494481] LNet: Added LNI 192.168.202.156@tcp125 [8/256/0/180] [13280.285454] LNet: Added LNI 192.168.202.156@tcp126 [8/256/0/180] [13281.223894] LNet: Added LNI 192.168.202.156@tcp127 [8/256/0/180] [13282.095941] LNet: Added LNI 192.168.202.156@tcp128 [8/256/0/180] [13283.254740] LNet: Added LNI 192.168.202.156@tcp129 [8/256/0/180] [13284.362142] LNet: Added LNI 192.168.202.156@tcp130 [8/256/0/180] [13285.009174] LNet: Added LNI 192.168.202.156@tcp131 [8/256/0/180] [13285.757889] LNet: Added LNI 192.168.202.156@tcp132 [8/256/0/180] [13286.679353] LNet: Added LNI 192.168.202.156@tcp133 [8/256/0/180] [13287.589295] LNet: Added LNI 192.168.202.156@tcp134 [8/256/0/180] [13288.316521] LNet: Added LNI 192.168.202.156@tcp135 [8/256/0/180] [13295.137615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13295.180875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13296.358098] Lustre: 434887:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [13296.425603] LNetError: 434908:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13296.438745] LNetError: Protocol error connecting to 192.168.202.156@tcp112 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13296.537149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13296.558462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13296.625534] Lustre: lustre-MDT0000: new disk, initializing [13296.714765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13296.732616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13297.442730] LNetError: 434909:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13297.478333] LNetError: Protocol error connecting to 192.168.202.156@tcp100 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13301.335401] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13311.624887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13311.820631] Lustre: 434905:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13311.876150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13311.890558] Lustre: Skipped 1 previous similar message [13311.987996] Lustre: lustre-MDT0001: new disk, initializing [13312.046535] Lustre: 435687:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [13312.060511] Lustre: 435687:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13312.209467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13312.248144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13312.264551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13314.491560] Lustre: 435796:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [13314.511534] Lustre: 435796:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13317.244713] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13324.877748] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13326.945439] LNetError: 432346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13326.957734] LNetError: Protocol error connecting to 192.168.202.156@tcp100 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13328.698085] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13333.105677] LNetError: 432343:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13333.114587] LNetError: Protocol error connecting to 192.168.202.156@tcp112 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13335.854641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13336.293621] Lustre: lustre-OST0000: new disk, initializing [13336.300920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13336.308988] Lustre: 436652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13336.443029] Lustre: 436657:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [13336.454699] Lustre: 436657:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13336.527114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13338.212797] LNetError: 434917:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13338.228905] LNetError: Protocol error connecting to 192.168.202.156@tcp100 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13343.087924] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13345.842492] Lustre: 437104:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.156@tcp: can't add NID 192.168.202.156@tcp130: rc = -75 [13345.862334] Lustre: 437104:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13345.949231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13345.957666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13346.100920] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13346.400779] LNetError: 434910:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13346.408366] LNetError: Protocol error connecting to 192.168.202.156@tcp112 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13349.343937] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13356.521873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13356.536974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13356.557930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13361.692170] Lustre: server umount lustre-OST0000 complete [13366.753495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13366.763578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13366.772980] Lustre: Skipped 2 previous similar messages [13366.784183] Lustre: Skipped 2 previous similar messages [13371.810595] Lustre: server umount lustre-MDT0000 complete [13371.875057] LustreError: 434921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13371.894506] LustreError: 434921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13372.012400] LNetError: 434908:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.156 [13372.045569] LNetError: 434908:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13372.053360] LNetError: Protocol error connecting to 192.168.202.156@tcp112 at host 192.168.202.156:988: is it running a compatible version of Lustre? [13372.069517] LNetError: Skipped 2 previous similar messages [13375.314822] LustreError: 437489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701041 with bad export cookie 10201072191157575751 [13375.317604] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13375.325517] LustreError: 437489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13375.608764] Lustre: server umount lustre-MDT0001 complete [13383.763744] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13386.362521] Key type lgssc unregistered [13386.642831] LNet: 438434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13386.649434] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13386.659302] LNet: Removed LNI 192.168.202.156@tcp [13387.679131] LNet: Removed LNI 192.168.202.156@tcp100 [13387.683173] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13387.686802] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13387.691353] LNet: Removed LNI 192.168.202.156@tcp101 [13388.703411] LNet: Removed LNI 192.168.202.156@tcp102 [13388.706646] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13388.709534] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13388.723222] LNet: Removed LNI 192.168.202.156@tcp103 [13389.727513] LNet: Removed LNI 192.168.202.156@tcp104 [13390.751186] LNet: Removed LNI 192.168.202.156@tcp105 [13390.755217] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13390.766601] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13390.788095] LNet: Removed LNI 192.168.202.156@tcp106 [13391.839196] LNet: Removed LNI 192.168.202.156@tcp107 [13392.863707] LNet: Removed LNI 192.168.202.156@tcp108 [13393.887192] LNet: Removed LNI 192.168.202.156@tcp109 [13394.911214] LNet: Removed LNI 192.168.202.156@tcp110 [13394.915762] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13394.950304] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13394.964515] LNet: Removed LNI 192.168.202.156@tcp111 [13395.999335] LNet: Removed LNI 192.168.202.156@tcp112 [13397.023465] LNet: Removed LNI 192.168.202.156@tcp113 [13398.047195] LNet: Removed LNI 192.168.202.156@tcp114 [13399.071413] LNet: Removed LNI 192.168.202.156@tcp115 [13400.095159] LNet: Removed LNI 192.168.202.156@tcp116 [13401.119867] LNet: Removed LNI 192.168.202.156@tcp117 [13402.207182] LNet: Removed LNI 192.168.202.156@tcp118 [13403.232428] LNet: Removed LNI 192.168.202.156@tcp119 [13403.237387] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13403.246722] LNetError: 438434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13403.258905] LNet: Removed LNI 192.168.202.156@tcp120 [13404.319148] LNet: Removed LNI 192.168.202.156@tcp121 [13404.321138] LNet: Removed LNI 192.168.202.156@tcp122 [13404.322409] LNet: Removed LNI 192.168.202.156@tcp123 [13404.329200] LNet: Removed LNI 192.168.202.156@tcp124 [13404.331683] LNet: Removed LNI 192.168.202.156@tcp125 [13404.334332] LNet: Removed LNI 192.168.202.156@tcp126 [13404.337273] LNet: Removed LNI 192.168.202.156@tcp127 [13404.341887] LNet: Removed LNI 192.168.202.156@tcp128 [13404.347476] LNet: Removed LNI 192.168.202.156@tcp129 [13404.352364] LNet: Removed LNI 192.168.202.156@tcp130 [13404.356370] LNet: Removed LNI 192.168.202.156@tcp131 [13404.360143] LNet: Removed LNI 192.168.202.156@tcp132 [13404.364875] LNet: Removed LNI 192.168.202.156@tcp133 [13404.368978] LNet: Removed LNI 192.168.202.156@tcp134 [13404.380965] LNet: Removed LNI 192.168.202.156@tcp135 [13405.336994] Key type .llcrypt unregistered [13405.338831] Key type ._llcrypt unregistered [13427.224342] Key type ._llcrypt registered [13427.225624] Key type .llcrypt registered [13427.311756] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [13440.989373] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13441.736394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13441.758186] alg: No test for adler32 (adler32-zlib) [13442.953326] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [13443.208316] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13444.927860] Key type lgssc registered [13446.284440] Lustre: Echo OBD driver; http://www.lustre.org/ [13480.853279] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:25:45 (1779701145) [13498.305111] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [13501.154441] Key type lgssc unregistered [13501.528725] LNet: 442467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13501.532707] LNetError: 442467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13501.546236] LNet: Removed LNI 192.168.202.156@tcp [13502.468238] Key type .llcrypt unregistered [13502.471698] Key type ._llcrypt unregistered [13518.977575] Key type ._llcrypt registered [13518.979804] Key type .llcrypt registered [13519.091115] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13520.185979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13520.316983] alg: No test for adler32 (adler32-zlib) [13521.409322] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [13521.676662] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [13523.335185] Key type lgssc registered [13524.445077] Lustre: Echo OBD driver; http://www.lustre.org/ [13533.655355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13533.702987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13535.024639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13535.067297] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13535.190372] Lustre: lustre-MDT0000: new disk, initializing [13535.320301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13535.355374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13539.802400] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13550.371673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13550.516619] Lustre: 443663:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13550.621566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13550.629473] Lustre: Skipped 1 previous similar message [13550.733625] Lustre: lustre-MDT0001: new disk, initializing [13550.856393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13550.884886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13550.891027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13555.475557] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13562.639686] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13566.124374] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13571.384824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13571.630437] Lustre: lustre-OST0000: new disk, initializing [13571.634286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13571.640150] Lustre: 445403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13571.712589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13576.917653] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13580.920234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13580.938620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13581.159298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13585.559114] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13588.703341] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing lnet_if_list [13594.059594] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13594.088640] LNet: Added LNI 192.168.202.156@tcp10 [8/256/0/180] [13599.580287] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13599.647870] LNet: Added LNI 192.168.202.156@tcp11 [8/256/0/180] [13604.841693] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13604.869865] LNet: Added LNI 192.168.202.156@tcp12 [8/256/0/180] [13609.577288] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13609.646434] LNet: Added LNI 192.168.202.156@tcp13 [8/256/0/180] [13614.422313] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13614.482297] LNet: Added LNI 192.168.202.156@tcp14 [8/256/0/180] [13618.686875] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13618.714338] LNet: Added LNI 192.168.202.156@tcp15 [8/256/0/180] [13623.221821] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13623.241649] LNet: Added LNI 192.168.202.156@tcp16 [8/256/0/180] [13627.870767] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13627.893405] LNet: Added LNI 192.168.202.156@tcp17 [8/256/0/180] [13632.609938] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13632.631804] LNet: Added LNI 192.168.202.156@tcp18 [8/256/0/180] [13637.547456] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13637.583533] LNet: Added LNI 192.168.202.156@tcp19 [8/256/0/180] [13643.524282] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13643.543748] LNet: Added LNI 192.168.202.156@tcp20 [8/256/0/180] [13649.278198] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13649.322134] LNet: Added LNI 192.168.202.156@tcp21 [8/256/0/180] [13654.767604] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13654.783846] LNet: Added LNI 192.168.202.156@tcp22 [8/256/0/180] [13659.864809] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13659.891097] LNet: Added LNI 192.168.202.156@tcp23 [8/256/0/180] [13665.545518] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13665.570653] LNet: Added LNI 192.168.202.156@tcp24 [8/256/0/180] [13671.981049] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13672.011401] LNet: Added LNI 192.168.202.156@tcp25 [8/256/0/180] [13677.803768] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13677.831125] LNet: Added LNI 192.168.202.156@tcp26 [8/256/0/180] [13683.481835] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13683.512461] LNet: Added LNI 192.168.202.156@tcp27 [8/256/0/180] [13688.821857] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13688.859430] LNet: Added LNI 192.168.202.156@tcp28 [8/256/0/180] [13694.464322] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13694.489304] LNet: Added LNI 192.168.202.156@tcp29 [8/256/0/180] [13701.837109] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13701.913679] LNet: Added LNI 192.168.202.156@tcp30 [8/256/0/180] [13708.340484] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13708.359649] LNet: Added LNI 192.168.202.156@tcp31 [8/256/0/180] [13715.395824] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13715.425454] LNet: Added LNI 192.168.202.156@tcp32 [8/256/0/180] [13721.623403] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13721.649055] LNet: Added LNI 192.168.202.156@tcp33 [8/256/0/180] [13728.202259] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13728.244506] LNet: Added LNI 192.168.202.156@tcp34 [8/256/0/180] [13734.740233] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13734.768659] LNet: Added LNI 192.168.202.156@tcp35 [8/256/0/180] [13741.716702] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13741.743890] LNet: Added LNI 192.168.202.156@tcp36 [8/256/0/180] [13748.598705] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13748.628871] LNet: Added LNI 192.168.202.156@tcp37 [8/256/0/180] [13755.131568] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13755.157387] LNet: Added LNI 192.168.202.156@tcp38 [8/256/0/180] [13761.537537] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13761.561323] LNet: Added LNI 192.168.202.156@tcp39 [8/256/0/180] [13768.655244] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13768.682432] LNet: Added LNI 192.168.202.156@tcp40 [8/256/0/180] [13776.197788] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13776.214301] LNet: Added LNI 192.168.202.156@tcp41 [8/256/0/180] [13783.785655] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13783.807455] LNet: Added LNI 192.168.202.156@tcp42 [8/256/0/180] [13791.511804] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13791.541961] LNet: Added LNI 192.168.202.156@tcp43 [8/256/0/180] [13800.196167] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13800.226047] LNet: Added LNI 192.168.202.156@tcp44 [8/256/0/180] [13808.075410] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13808.102784] LNet: Added LNI 192.168.202.156@tcp45 [8/256/0/180] [13815.642076] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13815.666888] LNet: Added LNI 192.168.202.156@tcp46 [8/256/0/180] [13823.684083] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13823.720107] LNet: Added LNI 192.168.202.156@tcp47 [8/256/0/180] [13830.646527] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13830.667029] LNet: Added LNI 192.168.202.156@tcp48 [8/256/0/180] [13838.355994] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13838.388708] LNet: Added LNI 192.168.202.156@tcp49 [8/256/0/180] [13847.429325] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13847.451810] LNet: Added LNI 192.168.202.156@tcp50 [8/256/0/180] [13868.008437] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13868.022651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13868.036107] Lustre: Skipped 2 previous similar messages [13868.039965] Lustre: Skipped 1 previous similar message [13871.689551] Lustre: server umount lustre-MDT0000 complete [13875.186730] LustreError: 445402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701541 with bad export cookie 13936520543652509653 [13875.188538] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13875.211330] LustreError: 445402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13875.486829] Lustre: server umount lustre-MDT0001 complete [13892.024296] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [13894.624156] Lustre: 442934:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779701544/real 1779701544] req@00000000d3528384 x1866151952019968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779701560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13894.648236] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13896.543196] Lustre: 442933:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779701546/real 1779701546] req@000000006cc59041 x1866151952020096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779701562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13899.743174] Lustre: 442935:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779701549/real 1779701549] req@00000000022cab71 x1866151952020224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779701565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13901.855272] Lustre: 442935:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779701551/real 1779701551] req@00000000f42b3bf4 x1866151952020480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779701567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13902.822880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13906.914418] Lustre: 442934:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779701556/real 1779701556] req@000000003d2cc384 x1866151952020608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779701572 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13920.362131] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc168779879578175 [13920.377323] Lustre: MGC192.168.202.156@tcp: Connection restored to 0@lo (at 0@lo) [13920.806245] LustreError: 445395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13920.818596] LustreError: 445395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13920.878482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13923.947323] LustreError: 445396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13923.959606] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13924.963331] LustreError: 445395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13926.997702] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13930.466413] LustreError: 445395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13935.589847] LustreError: 445396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13936.303899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13936.843714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13939.955359] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13941.943697] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [13947.837701] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13951.810332] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13971.937398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13971.948330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13971.964838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13971.968868] Lustre: Skipped 1 previous similar message [13972.963072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13972.973290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13976.039991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13976.048334] Lustre: Skipped 1 previous similar message [13977.523772] Lustre: server umount lustre-OST0000 complete [13983.203903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13983.210184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13983.218554] Lustre: Skipped 1 previous similar message [13983.220795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13987.819116] Lustre: server umount lustre-MDT0000 complete [13991.048428] LustreError: 443654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701657 with bad export cookie 13936520543652512117 [13991.059359] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13991.064208] LustreError: 443654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13991.438240] Lustre: server umount lustre-MDT0001 complete [14003.006270] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [14005.988747] Key type lgssc unregistered [14006.601993] LNet: 471355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14006.607941] LNetError: 471355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14006.615802] LNet: Removed LNI 192.168.202.156@tcp [14006.623340] LNet: Removed LNI 192.168.202.156@tcp10 [14006.626880] LNet: Removed LNI 192.168.202.156@tcp11 [14006.633909] LNet: Removed LNI 192.168.202.156@tcp12 [14006.635600] LNet: Removed LNI 192.168.202.156@tcp13 [14006.647509] LNet: Removed LNI 192.168.202.156@tcp14 [14006.650395] LNet: Removed LNI 192.168.202.156@tcp15 [14006.663170] LNet: Removed LNI 192.168.202.156@tcp16 [14006.667062] LNet: Removed LNI 192.168.202.156@tcp17 [14006.669062] LNet: Removed LNI 192.168.202.156@tcp18 [14006.676919] LNet: Removed LNI 192.168.202.156@tcp19 [14006.684246] LNet: Removed LNI 192.168.202.156@tcp20 [14006.688860] LNet: Removed LNI 192.168.202.156@tcp21 [14006.693843] LNet: Removed LNI 192.168.202.156@tcp22 [14006.699551] LNet: Removed LNI 192.168.202.156@tcp23 [14006.702553] LNet: Removed LNI 192.168.202.156@tcp24 [14006.709726] LNet: Removed LNI 192.168.202.156@tcp25 [14006.712446] LNet: Removed LNI 192.168.202.156@tcp26 [14006.714623] LNet: Removed LNI 192.168.202.156@tcp27 [14006.717069] LNet: Removed LNI 192.168.202.156@tcp28 [14006.719204] LNet: Removed LNI 192.168.202.156@tcp29 [14006.721409] LNet: Removed LNI 192.168.202.156@tcp30 [14006.722741] LNet: Removed LNI 192.168.202.156@tcp31 [14006.723931] LNet: Removed LNI 192.168.202.156@tcp32 [14006.725438] LNet: Removed LNI 192.168.202.156@tcp33 [14006.726739] LNet: Removed LNI 192.168.202.156@tcp34 [14006.727952] LNet: Removed LNI 192.168.202.156@tcp35 [14006.729993] LNet: Removed LNI 192.168.202.156@tcp36 [14006.732211] LNet: Removed LNI 192.168.202.156@tcp37 [14006.733961] LNet: Removed LNI 192.168.202.156@tcp38 [14006.740211] LNet: Removed LNI 192.168.202.156@tcp39 [14006.742334] LNet: Removed LNI 192.168.202.156@tcp40 [14006.743481] LNet: Removed LNI 192.168.202.156@tcp41 [14006.747976] LNet: Removed LNI 192.168.202.156@tcp42 [14006.750199] LNet: Removed LNI 192.168.202.156@tcp43 [14006.751972] LNet: Removed LNI 192.168.202.156@tcp44 [14006.753614] LNet: Removed LNI 192.168.202.156@tcp45 [14006.754928] LNet: Removed LNI 192.168.202.156@tcp46 [14006.759073] LNet: Removed LNI 192.168.202.156@tcp47 [14006.761721] LNet: Removed LNI 192.168.202.156@tcp48 [14006.764073] LNet: Removed LNI 192.168.202.156@tcp49 [14006.785146] LNet: Removed LNI 192.168.202.156@tcp50 [14008.042898] Key type .llcrypt unregistered [14008.052361] Key type ._llcrypt unregistered [14033.619627] Key type ._llcrypt registered [14033.623560] Key type .llcrypt registered [14033.742269] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [14049.580150] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [14051.040494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14051.158490] alg: No test for adler32 (adler32-zlib) [14052.806305] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [14053.195822] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [14054.927156] Key type lgssc registered [14057.003637] Lustre: Echo OBD driver; http://www.lustre.org/ [14091.050509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14091.075247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14092.396785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14092.463932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14092.603623] Lustre: lustre-MDT0000: new disk, initializing [14092.740028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14092.766046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14097.109447] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14106.981027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14107.166307] Lustre: 474532:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14107.200284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14107.202389] Lustre: Skipped 1 previous similar message [14107.274325] Lustre: lustre-MDT0001: new disk, initializing [14107.383587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14107.435824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14107.455821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14111.245177] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14117.915616] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14122.091867] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14131.536773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14131.840750] Lustre: *** cfs_fail_loc=90f, val=0*** [14132.061768] Lustre: lustre-OST0000: new disk, initializing [14132.072160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14132.095963] Lustre: 476320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14132.311473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14141.162914] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14142.174790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14142.211723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14142.461868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14150.077074] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14157.816812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14157.820619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14157.823317] LustreError: Skipped 1 previous similar message [14157.832435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14157.840910] Lustre: Skipped 1 previous similar message [14162.933390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14162.949706] Lustre: Skipped 1 previous similar message [14168.041355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14172.127587] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14172.528674] Lustre: server umount lustre-OST0000 complete [14178.272975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14178.279351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14178.293433] Lustre: Skipped 2 previous similar messages [14178.320490] Lustre: Skipped 2 previous similar messages [14183.394083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14183.402099] Lustre: Skipped 1 previous similar message [14191.075504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14191.352726] Lustre: server umount lustre-MDT0000 complete [14195.698227] LustreError: 475308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779701861 with bad export cookie 2348708094052415509 [14195.701126] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14195.704517] LustreError: 475308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14196.035402] Lustre: server umount lustre-MDT0001 complete [14206.589967] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [14209.376689] Key type lgssc unregistered [14209.699566] LNet: 478099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14209.720976] LNetError: 478099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14209.734141] LNet: Removed LNI 192.168.202.156@tcp [14210.698640] Key type .llcrypt unregistered [14210.701660] Key type ._llcrypt unregistered [14229.942885] Key type ._llcrypt registered [14229.945756] Key type .llcrypt registered [14230.037973] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [14230.991310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14231.077928] alg: No test for adler32 (adler32-zlib) [14232.268959] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [14232.574560] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [14234.351200] Key type lgssc registered [14235.381348] Lustre: Echo OBD driver; http://www.lustre.org/ [14243.604766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14243.625761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14245.568313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14250.347356] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14255.084254] LustreError: 479353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14255.119824] LustreError: 479353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14260.199163] LustreError: 479352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14260.519737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14260.864884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14265.675759] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14270.938755] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14275.225569] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14282.286262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14282.586391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14288.735486] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14294.594884] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14305.988390] loop0: detected capacity change from 0 to 204800000 [14306.206666] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14306.228157] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14306.252206] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14312.262444] loop0: detected capacity change from 0 to 204800000 [14312.459125] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14312.466418] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14312.492866] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14318.841591] loop0: detected capacity change from 0 to 204800000 [14325.080558] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14325.228206] Lustre: fs15246-OST0000: new disk, initializing [14325.234819] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14325.247826] Lustre: 482604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14325.328079] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14332.062443] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14340.499715] loop1: detected capacity change from 0 to 204800000 [14345.277165] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14345.339931] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14345.372328] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14345.438779] Lustre: fs15246-MDT0000: new disk, initializing [14345.516114] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14345.528626] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14345.647650] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14346.724119] Lustre: 483708:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006590e041 x1866152697441792/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1779702028 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14349.825278] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14355.821176] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14355.828675] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14355.882642] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14372.840176] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14372.863118] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14377.961837] LustreError: 479347:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14378.225951] Lustre: server umount fs15246-MDT0000 complete [14383.083705] LustreError: 483478:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14388.196773] LustreError: 479348:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14388.353378] Lustre: server umount fs15246-OST0000 complete [14403.553481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14403.555840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14403.565393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14403.594506] LustreError: Skipped 1 previous similar message [14408.677757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14408.697163] Lustre: Skipped 2 previous similar messages [14413.802961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14416.351739] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14416.621558] Lustre: server umount lustre-OST0000 complete [14424.037318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14424.053589] Lustre: Skipped 1 previous similar message [14424.058915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14424.064556] Lustre: Skipped 1 previous similar message [14427.814147] Lustre: server umount lustre-MDT0000 complete [14429.157617] LustreError: 479352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14429.175554] LustreError: 479352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14432.163917] LustreError: 479333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702098 with bad export cookie 2967188780737954633 [14432.164835] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14432.185953] LustreError: 479333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14432.652918] Lustre: server umount lustre-MDT0001 complete [14444.486807] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [14447.707741] Key type lgssc unregistered [14448.048659] LNet: 486772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14448.054927] LNetError: 486772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14448.065978] LNet: Removed LNI 192.168.202.156@tcp [14449.188176] Key type .llcrypt unregistered [14449.189934] Key type ._llcrypt unregistered [14470.197515] Key type ._llcrypt registered [14470.200450] Key type .llcrypt registered [14470.383943] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [14471.898408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14472.048337] alg: No test for adler32 (adler32-zlib) [14473.521326] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [14473.872980] LNet: Added LNI 192.168.202.156@tcp [8/256/0/180] [14475.679252] Key type lgssc registered [14476.939793] Lustre: Echo OBD driver; http://www.lustre.org/ [14519.295371] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [14528.364080] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [14570.350645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14570.401105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14571.676554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14571.720289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14571.834884] Lustre: lustre-MDT0000: new disk, initializing [14571.980731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14572.010740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14576.858753] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14590.224786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14590.473854] Lustre: 491193:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14590.566728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14590.574202] Lustre: Skipped 1 previous similar message [14590.759706] Lustre: lustre-MDT0001: new disk, initializing [14590.972828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14591.086866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14591.110462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14597.275564] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14605.922452] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14611.504219] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14621.082728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14621.478414] Lustre: lustre-OST0000: new disk, initializing [14621.483222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14621.492607] Lustre: 492933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14621.595737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14627.913082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14627.927030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14628.076555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14629.190912] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_default_debug -1 all [14637.506481] Lustre: DEBUG MARKER: oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14642.290884] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14642.523669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14646.962157] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14647.218140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14653.930807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14653.946067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14653.951927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14655.423182] Lustre: server umount lustre-OST0000 complete [14662.624954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14662.640043] LustreError: Skipped 1 previous similar message [14662.651181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14662.667190] Lustre: Skipped 1 previous similar message [14662.675950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14662.691815] Lustre: Skipped 1 previous similar message [14664.161586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14664.164546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14664.184509] Lustre: Skipped 1 previous similar message [14666.717957] Lustre: server umount lustre-MDT0000 complete [14671.656796] LustreError: 491185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779702337 with bad export cookie 16789957833660991358 [14671.668865] LustreError: MGC192.168.202.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14672.414559] Lustre: server umount lustre-MDT0001 complete [14700.627543] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing set_hostid [14711.205431] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing load_modules_local [14747.553647] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14404 sec ========= 05:46:51 (1779702411) [14749.609481] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:46:52 (1779702412) === [14754.112632] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:46:57 (1779702417) === [14796.788819] Lustre: DEBUG MARKER: oleg256-server.virtnet: executing unload_modules_local [14799.723614] Key type lgssc unregistered [14800.100089] LNet: 499380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14800.122688] LNetError: 499380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14801.191569] LNet: Removed LNI 192.168.202.156@tcp [14802.119151] Key type .llcrypt unregistered [14802.121412] Key type ._llcrypt unregistered