[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509040527 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003352] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009026] pid_max: default: 32768 minimum: 301 [ 0.010182] LSM: Security Framework initializing [ 0.011080] Yama: becoming mindful. [ 0.012055] SELinux: Initializing. [ 0.013119] *** VALIDATE selinux *** [ 0.021787] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025682] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029164] *** VALIDATE tmpfs *** [ 0.031290] *** VALIDATE proc *** [ 0.032283] *** VALIDATE cgroup *** [ 0.033015] *** VALIDATE cgroup2 *** [ 0.034342] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035176] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038041] Spectre V2 : User space: Vulnerable [ 0.039016] Speculative Store Bypass: Vulnerable [ 0.042871] debug: unmapping init [mem 0xffffffff97059000-0xffffffff97060fff] [ 0.045237] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046815] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048024] ... bit width: 48 [ 0.049021] ... generic registers: 4 [ 0.050020] ... value mask: 0000ffffffffffff [ 0.051024] ... max period: 00007fffffffffff [ 0.052026] ... fixed-purpose events: 3 [ 0.053020] ... event mask: 000000070000000f [ 0.054399] rcu: Hierarchical SRCU implementation. [ 0.056720] smp: Bringing up secondary CPUs ... [ 0.057752] x86: Booting SMP configuration: [ 0.058055] .... node #0, CPUs: #1 #2 #3 [ 0.062404] smp: Brought up 1 node, 4 CPUs [ 0.064024] smpboot: Max logical packages: 1 [ 0.065031] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.226704] node 0 deferred pages initialised in 159ms [ 0.231143] devtmpfs: initialized [ 0.233452] x86/mm: Memory block size: 128MB [ 0.238373] gcov: version magic: 0x41383552 [ 0.242380] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.248123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.251359] pinctrl core: initialized pinctrl subsystem [ 0.254241] [ 0.255013] ************************************************************* [ 0.260019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263016] ** ** [ 0.266018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.270018] ** ** [ 0.274019] ** This means that this kernel is built to expose internal ** [ 0.278020] ** IOMMU data structures, which may compromise security on ** [ 0.281014] ** your system. ** [ 0.284017] ** ** [ 0.287017] ** If you see this message and you are not debugging the ** [ 0.291017] ** kernel, report this immediately to your vendor! ** [ 0.294024] ** ** [ 0.297015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.300020] ************************************************************* [ 0.304828] NET: Registered protocol family 16 [ 0.307450] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.309065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316146] cpuidle: using governor menu [ 0.317671] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.318729] PCI: Using configuration type 1 for base access [ 0.320160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.334045] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.337018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.342044] cryptd: max_cpu_qlen set to 1000 [ 0.345298] ACPI: Added _OSI(Module Device) [ 0.347024] ACPI: Added _OSI(Processor Device) [ 0.350023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.354020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.359783] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.371398] ACPI: Interpreter enabled [ 0.372000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.374035] ACPI: Using IOAPIC for interrupt routing [ 0.378360] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.388000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.402499] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.408117] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.412026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.420095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.428354] acpiphp: Slot [2] registered [ 0.431217] acpiphp: Slot [5] registered [ 0.433587] acpiphp: Slot [6] registered [ 0.436115] acpiphp: Slot [7] registered [ 0.439165] acpiphp: Slot [8] registered [ 0.440130] acpiphp: Slot [9] registered [ 0.448243] acpiphp: Slot [10] registered [ 0.452294] acpiphp: Slot [3] registered [ 0.453141] acpiphp: Slot [4] registered [ 0.455280] acpiphp: Slot [11] registered [ 0.457169] acpiphp: Slot [12] registered [ 0.461228] acpiphp: Slot [13] registered [ 0.464159] acpiphp: Slot [14] registered [ 0.466155] acpiphp: Slot [15] registered [ 0.468226] acpiphp: Slot [16] registered [ 0.470171] acpiphp: Slot [17] registered [ 0.472246] acpiphp: Slot [18] registered [ 0.475147] acpiphp: Slot [19] registered [ 0.478342] acpiphp: Slot [20] registered [ 0.481140] acpiphp: Slot [21] registered [ 0.482135] acpiphp: Slot [22] registered [ 0.484236] acpiphp: Slot [23] registered [ 0.487149] acpiphp: Slot [24] registered [ 0.488144] acpiphp: Slot [25] registered [ 0.491166] acpiphp: Slot [26] registered [ 0.492000] acpiphp: Slot [27] registered [ 0.493148] acpiphp: Slot [28] registered [ 0.495126] acpiphp: Slot [29] registered [ 0.498135] acpiphp: Slot [30] registered [ 0.504168] acpiphp: Slot [31] registered [ 0.507095] PCI host bridge to bus 0000:00 [ 0.515031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.526059] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.533045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.546033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.557038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.571047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.577204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.583455] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.589039] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.616025] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.628064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.643027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.644029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.649025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.656427] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.663825] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.677000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.683094] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 18554 usecs [ 0.695023] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.711026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.734026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.751025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.774033] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.790043] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.806027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.830023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.842377] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.850020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.856020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.867020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.876000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.886019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.897019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.917023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.927361] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.946024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.967023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.997026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.015702] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.020033] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.031024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.049026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.066805] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.077020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.083021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.100022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.111124] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.114681] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.117632] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.119605] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.122475] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.127058] iommu: Default domain type: Passthrough [ 1.133663] SCSI subsystem initialized [ 1.135248] ACPI: bus type USB registered [ 1.141000] usbcore: registered new interface driver usbfs [ 1.152170] usbcore: registered new interface driver hub [ 1.159624] usbcore: registered new device driver usb [ 1.164152] pps_core: LinuxPPS API ver. 1 registered [ 1.165017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.175248] PTP clock support registered [ 1.178071] EDAC MC: Ver: 3.0.0 [ 1.182178] PCI: Using ACPI for IRQ routing [ 1.188804] NetLabel: Initializing [ 1.193027] NetLabel: domain hash size = 128 [ 1.196022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.205000] NetLabel: unlabeled traffic allowed by default [ 1.210052] vgaarb: loaded [ 1.212703] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.217020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.243436] clocksource: Switched to clocksource kvm-clock [ 1.533981] VFS: Disk quotas dquot_6.6.0 [ 1.535975] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.539593] *** VALIDATE ramfs *** [ 1.540640] *** VALIDATE hugetlbfs *** [ 1.543716] pnp: PnP ACPI init [ 1.546529] pnp: PnP ACPI: found 6 devices [ 1.574285] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.583529] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.589301] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.599640] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.605436] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.608828] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.616961] NET: Registered protocol family 2 [ 1.625605] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.642757] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.656940] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.669401] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.686509] TCP: Hash tables configured (established 65536 bind 65536) [ 1.696290] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.704972] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.721041] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.729929] NET: Registered protocol family 1 [ 1.737754] RPC: Registered named UNIX socket transport module. [ 1.740363] RPC: Registered udp transport module. [ 1.747508] RPC: Registered tcp transport module. [ 1.748870] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.755605] NET: Registered protocol family 44 [ 1.766362] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.775858] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.778443] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.784018] PCI: CLS 0 bytes, default 64 [ 1.792342] Unpacking initramfs... [ 5.339724] debug: unmapping init [mem 0xffff942b3cc54000-0xffff942b3ffbffff] [ 5.360714] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.374911] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.389284] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.615451] Initialise system trusted keyrings [ 6.618387] Key type blacklist registered [ 6.620725] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.638291] zbud: loaded [ 6.644419] *** VALIDATE nfs *** [ 6.647215] *** VALIDATE nfs4 *** [ 6.649776] pstore: using deflate compression [ 6.653699] Platform Keyring initialized [ 6.922391] NET: Registered protocol family 38 [ 6.924305] Key type asymmetric registered [ 6.925735] Asymmetric key parser 'x509' registered [ 6.936381] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.943904] io scheduler mq-deadline registered [ 6.945753] io scheduler kyber registered [ 6.947499] io scheduler bfq registered [ 6.949473] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.952893] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.955819] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.958374] ACPI: Power Button [PWRF] [ 6.965847] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.972849] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.988051] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.002562] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.020299] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.052129] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.084446] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.092283] Non-volatile memory driver v1.3 [ 7.095383] Linux agpgart interface v0.103 [ 7.142222] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 7.147516] vda: detected capacity change from 0 to 69914624 [ 7.211510] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.222730] vdb: detected capacity change from 0 to 1073741824 [ 7.315338] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.330218] vdc: detected capacity change from 0 to 2621440000 [ 7.395359] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.401155] vdd: detected capacity change from 0 to 2621440000 [ 7.459911] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.475533] vde: detected capacity change from 0 to 4294967296 [ 7.554232] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.562436] vdf: detected capacity change from 0 to 4294967296 [ 7.596630] libphy: Fixed MDIO Bus: probed [ 7.636794] usbcore: registered new interface driver usbserial_generic [ 7.645239] usbserial: USB Serial support registered for generic [ 7.653385] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.673674] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.689130] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.699188] mousedev: PS/2 mouse device common for all mice [ 7.714526] rtc_cmos 00:05: RTC can wake from S4 [ 7.719968] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.722316] rtc_cmos 00:05: registered as rtc0 [ 7.741709] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.773795] intel_pstate: CPU model not supported [ 7.784781] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.796515] hid: raw HID events driver (C) Jiri Kosina [ 7.810507] usbcore: registered new interface driver usbhid [ 7.813591] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.813726] usbhid: USB HID core driver [ 7.818142] drop_monitor: Initializing network drop monitor service [ 7.818326] Initializing XFRM netlink socket [ 7.818730] NET: Registered protocol family 10 [ 7.840418] Segment Routing with IPv6 [ 7.840528] NET: Registered protocol family 17 [ 7.842957] mpls_gso: MPLS GSO support [ 7.892540] RAS: Correctable Errors collector initialized. [ 7.903700] AVX version of gcm_enc/dec engaged. [ 7.910983] AES CTR mode by8 optimization enabled [ 8.302133] sched_clock: Marking stable (8302095798, 0)->(9490540317, -1188444519) [ 8.307977] registered taskstats version 1 [ 8.312761] Loading compiled-in X.509 certificates [ 8.319736] zswap: loaded using pool lzo/zbud [ 8.450315] Key type big_key registered [ 8.542293] Key type encrypted registered [ 8.545893] ima: No TPM chip found, activating TPM-bypass! [ 8.561941] ima: Allocated hash algorithm: sha1 [ 8.564319] ima: No architecture policies found [ 8.566661] evm: Initialising EVM extended attributes: [ 8.571995] evm: security.selinux [ 8.583870] evm: security.ima [ 8.591640] evm: security.capability [ 8.602641] evm: HMAC attrs: 0x1 [ 8.617680] rtc_cmos 00:05: setting system clock to 2026-06-11 22:35:15 UTC (1781217315) [ 8.655686] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 8.664810] debug: unmapping init [mem 0xffffffff96d82000-0xffffffff97058fff] [ 8.674498] Write protecting the kernel read-only data: 28672k [ 8.683044] debug: unmapping init [mem 0xffffffff95403000-0xffffffff955fffff] [ 8.697156] debug: unmapping init [mem 0xffffffff95d14000-0xffffffff95dfffff] [ 8.836217] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.848471] systemd[1]: Detected virtualization kvm. [ 8.851181] systemd[1]: Detected architecture x86-64. [ 8.854215] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.902553] systemd[1]: No hostname configured. [ 8.906348] systemd[1]: Set hostname to . [ 8.912842] random: systemd: uninitialized urandom read (16 bytes read) [ 8.917082] systemd[1]: Initializing machine ID from random generator. [ 9.041781] random: ln: uninitialized urandom read (6 bytes read) [ 9.298810] random: systemd: uninitialized urandom read (16 bytes read) [ 9.302739] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.319989] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.331635] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.081967] device-mapper: uevent: version 1.0.3 [ 11.086261] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.715415] virtio_net virtio0 ens2: renamed from eth0 [ 12.762738] random: fast init done [ 12.987964] scsi host0: ata_piix [ 13.044232] scsi host1: ata_piix [ 13.048481] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.053920] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.132518] random: crng init done [ 18.137866] random: 7 urandom warning(s) missed due to ratelimiting [ 20.464922] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.566320] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ 25.074057] hrtimer: interrupt took 3123359 ns [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.142490] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.986249] SELinux: Disabled at runtime. [ 27.154698] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.196466] systemd[1]: Detected virtualization kvm. [ 27.198087] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.821562] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.840206] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.879626] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.891143] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.895861] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.933471] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.961309] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ 29.552938] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.772779] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.052769] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.483897] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.869442] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.985039] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 38.496491] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 39.548429] NFS: Registering the id_resolver key type [ 39.550958] Key type id_resolver registered [ 39.552980] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-server login: [ 105.162029] libcfs: loading out-of-tree module taints kernel. [ 105.255256] Key type ._llcrypt registered [ 105.259234] Key type .llcrypt registered [ 105.435857] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 126.296973] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 128.162281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.187468] alg: No test for adler32 (adler32-zlib) [ 129.684648] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 130.685419] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 132.608256] Key type lgssc registered [ 134.359153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 154.785443] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 202.516631] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 219.652688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 219.703287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.966344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.010347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.106910] Lustre: lustre-MDT0000: new disk, initializing [ 221.191455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.214343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 225.458324] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.372388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.518128] Lustre: 6516:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 240.601192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 240.609764] Lustre: Skipped 1 previous similar message [ 240.755255] Lustre: lustre-MDT0001: new disk, initializing [ 240.874877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.932812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 240.955114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.606390] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.518646] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.346820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.624702] Lustre: lustre-OST0000: new disk, initializing [ 260.629726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.642635] Lustre: 8419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.702066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 265.784293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 265.806402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 265.888608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 267.534886] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.516664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.655823] Lustre: lustre-OST0001: new disk, initializing [ 285.669866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 285.689762] Lustre: 9475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 285.850677] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 293.520919] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.980928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 294.005586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 294.047479] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 306.343489] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.369173] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 322.181580] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing check_logdir /tmp/testlogs/ [ 327.082590] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing yml_node [ 331.927514] Lustre: DEBUG MARKER: Client: 2.17.53.75 [ 334.531957] Lustre: DEBUG MARKER: MDS: 2.17.53.75 [ 336.884808] Lustre: DEBUG MARKER: OSS: 2.17.53.75 [ 338.896974] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 18:40:43 EDT 2026 [ 359.678847] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 361.260765] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 379.371248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 379.381684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.393425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.414542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 381.418407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.433048] Lustre: Skipped 2 previous similar messages [ 384.370818] Lustre: server umount lustre-MDT0000 complete [ 388.255901] LustreError: 8430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217695 with bad export cookie 1490094333533829920 [ 388.260333] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 388.272482] LustreError: 8430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 391.656081] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.656446] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 391.656875] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.656880] Lustre: Skipped 3 previous similar messages [ 391.672521] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 391.685679] Lustre: Skipped 1 previous similar message [ 393.696064] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.696680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 393.722259] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 393.751124] Lustre: Skipped 2 previous similar messages [ 394.683382] Lustre: server umount lustre-MDT0001 complete [ 399.308785] Lustre: server umount lustre-OST0000 complete [ 403.586986] Lustre: server umount lustre-OST0001 complete [ 410.156473] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 421.732093] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 461.120925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.542435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 461.578901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 461.695207] Lustre: lustre-MDT0000: new disk, initializing [ 461.784868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.834019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 466.760875] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 478.251129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.409575] Lustre: 14995:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 478.521938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 478.529546] Lustre: Skipped 1 previous similar message [ 478.665362] Lustre: lustre-MDT0001: new disk, initializing [ 478.772101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 478.801662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 478.814304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 483.180359] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 491.410757] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.921655] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.677631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.057312] Lustre: lustre-OST0000: new disk, initializing [ 504.061668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 504.066673] Lustre: 16737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 504.161640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 505.521348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 505.533367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 505.715761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 512.465818] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 523.238526] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 528.975229] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 529.282892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 534.755386] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 534.946977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 541.664669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.671207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.677378] LustreError: Skipped 1 previous similar message [ 541.688768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 541.696884] Lustre: Skipped 1 previous similar message [ 541.729776] Lustre: Skipped 1 previous similar message [ 544.029921] Lustre: server umount lustre-OST0000 complete [ 550.378041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 550.391098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.419398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.958495] Lustre: server umount lustre-MDT0000 complete [ 557.034039] LustreError: 15006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.062651] LustreError: 15006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 559.823935] LustreError: 14987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217866 with bad export cookie 1490094333533831453 [ 559.841558] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.742833] Lustre: server umount lustre-MDT0001 complete [ 567.371636] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:44:31 (1781217871) [ 570.934322] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 573.447946] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:44:37 (1781217877) [ 580.971092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.546205] LustreError: 18999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.652250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.793075] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 586.723097] LustreError: 19000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.824058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.323764] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 610.549626] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 615.187213] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 622.992030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 623.108131] LustreError: 20592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.131493] LustreError: 20592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 629.837129] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 636.018041] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 648.161144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 648.180359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.189030] LustreError: Skipped 1 previous similar message [ 648.189815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 648.189819] Lustre: Skipped 2 previous similar messages [ 648.228558] Lustre: Skipped 2 previous similar messages [ 650.340427] Lustre: server umount lustre-OST0000 complete [ 661.427994] Lustre: server umount lustre-MDT0000 complete [ 663.522738] LustreError: 19711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.538272] LustreError: 19711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 666.024374] LustreError: 18982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781217972 with bad export cookie 1490094333533832335 [ 666.034737] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.040059] LustreError: 18982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 666.551709] Lustre: server umount lustre-MDT0001 complete [ 683.037634] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 688.308696] Key type lgssc unregistered [ 688.499969] LNet: 22185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.506995] LNetError: 22185:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.519243] LNet: Removed LNI 192.168.202.136@tcp [ 689.407317] Key type .llcrypt unregistered [ 689.410664] Key type ._llcrypt unregistered [ 719.864169] Key type ._llcrypt registered [ 719.869525] Key type .llcrypt registered [ 720.064201] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 721.479984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.556641] alg: No test for adler32 (adler32-zlib) [ 722.982948] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 723.283251] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 725.031279] Key type lgssc registered [ 725.917836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 734.996669] loop: module loaded [ 735.124479] loop0: detected capacity change from 0 to 204800000 [ 735.475674] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 735.486828] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 735.934973] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.682867] loop0: detected capacity change from 0 to 204800000 [ 742.845444] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.866476] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.884092] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 748.293846] loop0: detected capacity change from 0 to 204800000 [ 754.142411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 754.174174] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.529222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 755.615735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 755.724035] Lustre: lustre-MDT0000: new disk, initializing [ 755.872448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 755.929217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 762.139369] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 772.785958] loop1: detected capacity change from 0 to 204800000 [ 778.851273] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 779.145469] Lustre: lustre-OST07c6: new disk, initializing [ 779.150392] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 779.156454] Lustre: Skipped 1 previous similar message [ 779.161053] Lustre: 25342:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 779.228346] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 785.929078] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 789.128356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 789.139781] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 789.183826] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 792.315108] Lustre: 24348:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 792.331591] Lustre: 24348:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 792.339980] Lustre: 24348:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 792.352187] Lustre: 24348:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 792.363366] Lustre: 24348:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 792.379586] Lustre: 24348:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 805.355350] Lustre: 24348:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 805.365149] Lustre: 24348:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 805.385526] Lustre: 24348:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 805.400750] Lustre: 24348:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 805.415501] Lustre: 24348:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 805.422205] Lustre: 24348:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.653479] Lustre: 24348:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 810.670020] Lustre: 24348:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 810.680241] Lustre: 24348:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 810.688942] Lustre: 24348:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.697587] Lustre: 24348:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 810.707105] Lustre: 24348:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.712317] Lustre: 24348:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 810.717478] Lustre: 24348:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.724124] Lustre: 24348:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 810.728678] Lustre: 24348:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 810.732747] Lustre: 24348:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 810.736970] Lustre: 24348:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.957744] Lustre: 25810:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 813.973876] Lustre: 25810:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.988863] Lustre: 25810:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 813.996846] Lustre: 25810:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.007820] Lustre: 25810:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 814.015309] Lustre: 25810:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.029873] Lustre: 25810:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 814.041027] Lustre: 25810:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.055696] Lustre: 25810:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 814.071629] Lustre: 25810:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.078160] Lustre: 25810:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.086768] Lustre: 25810:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.788348] Lustre: 25810:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 818.805727] Lustre: 25810:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 818.821220] Lustre: 25810:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 818.834241] Lustre: 25810:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.843400] Lustre: 25810:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 818.850390] Lustre: 25810:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.862096] Lustre: 25810:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 818.871348] Lustre: 25810:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.878882] Lustre: 25810:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 818.886141] Lustre: 25810:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.896290] Lustre: 25810:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 818.902828] Lustre: 25810:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 831.395385] Lustre: 24346:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 831.412711] Lustre: 24346:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 831.421802] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 831.429984] Lustre: 24346:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 831.437513] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 831.442920] Lustre: 24346:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 831.448702] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 831.454711] Lustre: 24346:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 831.460840] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 831.472667] Lustre: 24346:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 831.485421] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 831.499769] Lustre: 24346:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 881.124965] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 881.136571] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 881.138687] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.240982] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 886.974201] Lustre: server umount lustre-OST07c6 complete [ 898.894823] Lustre: server umount lustre-MDT0000 complete [ 918.193638] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 920.932739] Key type lgssc unregistered [ 921.202448] LNet: 27865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.206829] LNetError: 27865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.281721] LNet: Removed LNI 192.168.202.136@tcp [ 923.245167] Key type .llcrypt unregistered [ 923.248559] Key type ._llcrypt unregistered [ 945.285528] Key type ._llcrypt registered [ 945.288441] Key type .llcrypt registered [ 945.436172] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 946.709218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 946.908909] alg: No test for adler32 (adler32-zlib) [ 948.130739] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 948.295602] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 949.935228] Key type lgssc registered [ 951.137447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.773373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.802316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.459896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.335127] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 971.759596] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.795562] LustreError: 29119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 976.881620] LustreError: 29118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.334072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.942158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.459869] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 991.042513] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 995.335622] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1003.877867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1004.509343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.625080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1012.419889] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1020.484987] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1043.425099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1043.431780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.443080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.449770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1044.451656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1044.455193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.481173] Lustre: Skipped 1 previous similar message [ 1049.574397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.592256] Lustre: Skipped 1 previous similar message [ 1054.695234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.710530] Lustre: Skipped 1 previous similar message [ 1057.247672] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1057.475550] Lustre: server umount lustre-OST0000 complete [ 1064.929134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.934660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1064.950042] Lustre: Skipped 1 previous similar message [ 1069.134321] Lustre: server umount lustre-MDT0000 complete [ 1070.063194] LustreError: 31273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.095654] LustreError: 31273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1074.084189] LustreError: 29099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218380 with bad export cookie 8899764544097462287 [ 1074.094514] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.110059] LustreError: 29099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1074.644838] Lustre: server umount lustre-MDT0001 complete [ 1088.534467] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1091.386837] Key type lgssc unregistered [ 1091.697376] LNet: 32505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.703659] LNetError: 32505:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.732145] LNet: Removed LNI 192.168.202.136@tcp [ 1092.679395] Key type .llcrypt unregistered [ 1092.683956] Key type ._llcrypt unregistered [ 1113.792690] Key type ._llcrypt registered [ 1113.799128] Key type .llcrypt registered [ 1113.890781] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1115.236842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.262173] alg: No test for adler32 (adler32-zlib) [ 1116.410827] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1116.790664] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1118.559538] Key type lgssc registered [ 1120.384569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1130.515568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1130.563817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.600052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.982905] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1142.763839] LustreError: 33758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.786892] LustreError: 33758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1147.871962] LustreError: 33759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.705515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.034125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.058693] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1158.846760] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1162.566830] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1169.480538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.985229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.003709] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1177.158379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1182.210819] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1197.546685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.553545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.557761] Lustre: Skipped 1 previous similar message [ 1197.574802] Lustre: Skipped 1 previous similar message [ 1201.641253] Lustre: server umount lustre-OST0000 complete [ 1207.783230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.808500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.819100] Lustre: Skipped 1 previous similar message [ 1212.167492] Lustre: server umount lustre-MDT0000 complete [ 1212.909501] LustreError: 33758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.930157] LustreError: 33758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1216.059114] LustreError: 33739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218522 with bad export cookie 2225561166618970583 [ 1216.065455] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.066019] LustreError: 33739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1216.409780] Lustre: server umount lustre-MDT0001 complete [ 1229.132836] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1231.772531] Key type lgssc unregistered [ 1232.019940] LNet: 36958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.024704] LNetError: 36958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.038742] LNet: Removed LNI 192.168.202.136@tcp [ 1232.867716] Key type .llcrypt unregistered [ 1232.870887] Key type ._llcrypt unregistered [ 1261.440988] Key type ._llcrypt registered [ 1261.450468] Key type .llcrypt registered [ 1261.584163] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1262.604258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.640166] alg: No test for adler32 (adler32-zlib) [ 1263.783339] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1264.047785] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1265.767263] Key type lgssc registered [ 1267.142902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1277.430524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1277.470415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.213228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.863351] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1289.698028] LustreError: 38361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.722380] LustreError: 38361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1294.827891] LustreError: 38362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.192678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.910365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1304.112554] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1310.885731] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1315.691735] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1324.458235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1324.747993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1331.598151] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1338.842089] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1349.316857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1358.304448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1358.321186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.334772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.840043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.842974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.882066] Lustre: Skipped 1 previous similar message [ 1361.420358] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1361.425902] Lustre: Skipped 2 previous similar messages [ 1362.798124] Lustre: server umount lustre-MDT0000 complete [ 1364.962550] LustreError: 38361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1364.984202] LustreError: 38361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1366.600188] LustreError: 38342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218673 with bad export cookie 16832196600567027733 [ 1366.615216] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.617033] LustreError: 38342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1370.087384] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.091598] LustreError: 38357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.111278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1370.124069] LustreError: 38357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1373.119577] Lustre: server umount lustre-MDT0001 complete [ 1402.335137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1402.681709] Lustre: server umount lustre-OST0000 complete [ 1422.540222] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1426.552577] Key type lgssc unregistered [ 1427.046818] LNet: 41891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.053859] LNetError: 41891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.074552] LNet: Removed LNI 192.168.202.136@tcp [ 1428.485793] Key type .llcrypt unregistered [ 1428.494828] Key type ._llcrypt unregistered [ 1456.943180] Key type ._llcrypt registered [ 1456.946683] Key type .llcrypt registered [ 1457.056799] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1458.240235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1458.326520] alg: No test for adler32 (adler32-zlib) [ 1459.489744] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1459.727501] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1461.415302] Key type lgssc registered [ 1462.698741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1470.890638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1470.906656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.522510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.177523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1482.728398] LustreError: 43291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.768823] LustreError: 43291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1487.463464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.813117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.267604] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1499.981243] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1504.476866] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1513.183526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1513.824099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.968676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1522.619508] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1530.118794] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1538.535524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1538.548651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.573067] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1542.630955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.633410] Lustre: Skipped 1 previous similar message [ 1542.636896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.501906] Lustre: server umount lustre-OST0000 complete [ 1568.227393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.232227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.243080] Lustre: Skipped 1 previous similar message [ 1568.262552] Lustre: Skipped 3 previous similar messages [ 1571.503288] Lustre: server umount lustre-MDT0000 complete [ 1573.347914] LustreError: 43290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.368187] LustreError: 43290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1575.485972] LustreError: 43273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218882 with bad export cookie 2742497708356458004 [ 1575.510462] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1575.517151] LustreError: 43273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1582.151863] Lustre: server umount lustre-MDT0001 complete [ 1597.497442] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1600.832914] Key type lgssc unregistered [ 1601.188053] LNet: 46657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.193533] LNetError: 46657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.210189] LNet: Removed LNI 192.168.202.136@tcp [ 1602.217773] Key type .llcrypt unregistered [ 1602.219250] Key type ._llcrypt unregistered [ 1631.218142] Key type ._llcrypt registered [ 1631.222918] Key type .llcrypt registered [ 1631.341077] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1633.063259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.108861] alg: No test for adler32 (adler32-zlib) [ 1634.142512] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1634.368206] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1636.000662] Key type lgssc registered [ 1637.157618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1645.741646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1645.779028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.723803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.234214] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1657.827675] LustreError: 48057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.843906] LustreError: 48057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1662.946081] LustreError: 48058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1664.094295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.378888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1669.951622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1676.572747] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1681.122293] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1690.177389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.809250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.873545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1697.809198] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1705.202506] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1714.435935] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1718.262778] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1730.851385] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1119h03m54s [ 1732.883733] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1736.163943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.172956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.202896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.255457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1737.259546] Lustre: Skipped 2 previous similar messages [ 1739.271249] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1739.281623] Lustre: Skipped 3 previous similar messages [ 1742.392354] Lustre: server umount lustre-MDT0000 complete [ 1743.339823] LustreError: 49171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1743.367766] LustreError: 49171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1745.354781] LustreError: 48053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1746.942432] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1749.480346] LustreError: 48058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1749.525184] LustreError: 48058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1755.192610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.327667] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1755.592805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.676034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1756.606336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1761.663551] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1765.935743] Lustre: DEBUG MARKER: Wait for df (36106) ... [ 1767.998982] Lustre: DEBUG MARKER: done [ 1776.097253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1776.104839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.119332] Lustre: Skipped 2 previous similar messages [ 1776.133225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.139225] Lustre: Skipped 1 previous similar message [ 1779.493967] Lustre: server umount lustre-OST0000 complete [ 1787.361603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1787.363219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.368257] LustreError: Skipped 1 previous similar message [ 1787.369890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.394181] Lustre: Skipped 2 previous similar messages [ 1787.412393] Lustre: Skipped 3 previous similar messages [ 1798.111523] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1798.373975] Lustre: server umount lustre-MDT0000 complete [ 1800.683275] LustreError: 48057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.694617] LustreError: 48057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1802.805067] LustreError: 48040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219109 with bad export cookie 2798640344698897255 [ 1802.811508] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.818278] LustreError: 48040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1803.230896] Lustre: server umount lustre-MDT0001 complete [ 1818.413703] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1821.612863] Key type lgssc unregistered [ 1821.890139] LNet: 52612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.901875] LNetError: 52612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.919038] LNet: Removed LNI 192.168.202.136@tcp [ 1823.106387] Key type .llcrypt unregistered [ 1823.109215] Key type ._llcrypt unregistered [ 1875.358350] Key type ._llcrypt registered [ 1875.363266] Key type .llcrypt registered [ 1875.493613] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1877.529400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1877.569916] alg: No test for adler32 (adler32-zlib) [ 1878.844987] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 1879.432083] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1881.199274] Key type lgssc registered [ 1882.753670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.406965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.438605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.011700] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1896.070555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1896.440697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.561315] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1916.840525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.177791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1917.254121] Lustre: 54809:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1917.714499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.579658] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1933.313886] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1939.131591] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1948.201948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.600093] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1949.049698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.477903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1957.123587] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1964.135731] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1971.450340] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1119h07m56s [ 1973.750416] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1983.742423] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1987.512037] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1987.739268] LustreError: 55947:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2008.077794] Lustre: lustre-MDT0000: Client db9835d0-145b-4860-8af4-2a33b4c08369 (at 192.168.202.36@tcp) reconnecting [ 2027.791274] LustreError: 55947:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2030.752749] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2034.552105] Lustre: DEBUG MARKER: done [ 2043.360965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.372158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.388382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2043.874854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.901279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.923099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2048.992362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.010306] Lustre: Skipped 1 previous similar message [ 2049.517529] Lustre: server umount lustre-OST0000 complete [ 2056.160247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2056.172637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.193491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.322794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.344780] Lustre: Skipped 4 previous similar messages [ 2061.329451] Lustre: server umount lustre-MDT0000 complete [ 2066.321597] LustreError: 55547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219373 with bad export cookie 11848558667446206764 [ 2066.336411] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.345438] LustreError: 55547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2066.944710] Lustre: server umount lustre-MDT0001 complete [ 2086.354500] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2090.334168] Key type lgssc unregistered [ 2090.692710] LNet: 58808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.698313] LNetError: 58808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.714114] LNet: Removed LNI 192.168.202.136@tcp [ 2091.755239] Key type .llcrypt unregistered [ 2091.760993] Key type ._llcrypt unregistered [ 2146.853946] Key type ._llcrypt registered [ 2146.856211] Key type .llcrypt registered [ 2146.927993] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2148.490307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.533313] alg: No test for adler32 (adler32-zlib) [ 2149.634784] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2149.939037] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2151.655207] Key type lgssc registered [ 2152.821794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2156.222035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2156.243664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.586440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2157.624692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2157.836773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.767780] Lustre: Failing over lustre-MDT0000 [ 2159.303902] Lustre: server umount lustre-MDT0000 complete [ 2168.699077] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:11:12 (1781219472) [ 2177.879562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.663826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.757778] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2198.982511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.128867] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2199.162454] Lustre: 61594:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2199.535590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2205.403560] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2212.182568] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2217.450436] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2225.503477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2225.747646] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2225.916220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.054854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2234.019556] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2242.357759] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2249.505693] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2258.913091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.915938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.932799] Lustre: Skipped 1 previous similar message [ 2258.949643] Lustre: Skipped 2 previous similar messages [ 2264.032477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.050327] Lustre: Skipped 2 previous similar messages [ 2269.153598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.225777] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2272.487836] Lustre: server umount lustre-MDT0000 complete [ 2276.362809] LustreError: 62323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219583 with bad export cookie 5736826324718002659 [ 2276.370354] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2276.372359] LustreError: 62323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2276.827490] Lustre: server umount lustre-MDT0001 complete [ 2280.845276] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2285.808341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2295.289431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.775372] Lustre: 60518:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219586/real 1781219586] req@00000000f0569980 x1867743972340096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2295.818180] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.830780] Lustre: Skipped 1 previous similar message [ 2297.631148] Lustre: 60517:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219588/real 1781219588] req@00000000c01440bf x1867743972340224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2300.895534] Lustre: 60515:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219591/real 1781219591] req@00000000e7a173d1 x1867743972340352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2301.984834] LustreError: 60514:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f0569980 x1867743972340992/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2302.497834] LustreError: 63219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.504531] LustreError: 63219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2302.652277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.782038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2303.583262] Lustre: 60515:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219593/real 1781219593] req@000000009de46a35 x1867743972340608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2304.691527] LustreError: 63220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.693637] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2305.765088] LustreError: 63219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.805247] LustreError: 63219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2307.289857] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2308.575158] Lustre: 60518:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219598/real 1781219598] req@0000000020fd7347 x1867743972340736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2311.153128] LustreError: 65028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.189772] LustreError: 65028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2316.265698] LustreError: 65028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.296523] LustreError: 65028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2317.740168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.172176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2322.676192] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2323.439388] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2329.853077] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2335.012645] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2349.029867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.030936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.033936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2349.051443] Lustre: Skipped 1 previous similar message [ 2349.060824] Lustre: Skipped 2 previous similar messages [ 2353.056232] Lustre: server umount lustre-MDT0000 complete [ 2354.152424] LustreError: 65029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.188994] LustreError: 65029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2357.301609] LustreError: 61586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219664 with bad export cookie 5736826324718004934 [ 2357.303751] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.310880] LustreError: 61586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2357.604215] Lustre: server umount lustre-MDT0001 complete [ 2365.477748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2375.647090] Lustre: 60517:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219666/real 1781219666] req@000000001fd0b344 x1867743972369408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219682 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2375.696564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.721368] Lustre: Skipped 1 previous similar message [ 2379.348708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.345924] LustreError: 65746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.399786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.463776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2388.925614] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2399.004726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.726591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2405.190511] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2413.108389] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2419.072352] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2430.436686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2430.438394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.456158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.456166] Lustre: Skipped 2 previous similar messages [ 2430.484093] Lustre: Skipped 2 previous similar messages [ 2433.640773] Lustre: server umount lustre-MDT0000 complete [ 2435.556976] LustreError: 67485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.585096] LustreError: 67485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2438.941404] LustreError: 61587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219745 with bad export cookie 5736826324718006313 [ 2438.956536] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.843720] Lustre: server umount lustre-MDT0001 complete [ 2448.056839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2452.949185] Lustre: DEBUG MARKER: files compared the same [ 2465.393955] Lustre: server umount lustre-OST0000 complete [ 2488.067901] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2490.793519] Key type lgssc unregistered [ 2491.020732] LNet: 70535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.033894] LNetError: 70535:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.072925] LNet: Removed LNI 192.168.202.136@tcp [ 2493.317931] Key type .llcrypt unregistered [ 2493.321120] Key type ._llcrypt unregistered [ 2526.022690] Key type ._llcrypt registered [ 2526.025679] Key type .llcrypt registered [ 2526.169953] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2528.084885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2528.130231] alg: No test for adler32 (adler32-zlib) [ 2529.303687] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2529.570467] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2531.319182] Key type lgssc registered [ 2533.390959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2542.588387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2542.616540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.269688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.039627] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2554.850666] LustreError: 71936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.880217] LustreError: 71936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2559.844957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.968089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2560.168143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2565.604214] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2572.726883] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2577.269683] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2585.366428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2585.627633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.750813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2593.222982] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2601.303835] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2619.942743] Lustre: server umount lustre-OST0000 complete [ 2625.504412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.509979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2625.527777] Lustre: Skipped 1 previous similar message [ 2625.540163] Lustre: Skipped 1 previous similar message [ 2630.624331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.638370] Lustre: Skipped 1 previous similar message [ 2635.745955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.772269] Lustre: Skipped 1 previous similar message [ 2635.881666] Lustre: server umount lustre-MDT0000 complete [ 2636.768092] Lustre: 71250:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219927/real 1781219927] req@0000000033d69ef1 x1867744370781824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781219943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2636.772218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.834372] Lustre: 71250:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2636.861960] LustreError: 71931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.234953] LustreError: 71919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219948 with bad export cookie 9990871424227065720 [ 2641.243580] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.248236] LustreError: 71919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2641.983416] Lustre: server umount lustre-MDT0001 complete [ 2656.257926] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2659.354328] Key type lgssc unregistered [ 2659.762763] LNet: 75136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.773317] LNetError: 75136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.799775] LNet: Removed LNI 192.168.202.136@tcp [ 2660.905760] Key type .llcrypt unregistered [ 2660.913826] Key type ._llcrypt unregistered [ 2692.606244] Key type ._llcrypt registered [ 2692.614301] Key type .llcrypt registered [ 2692.733994] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2694.038838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2694.185515] alg: No test for adler32 (adler32-zlib) [ 2695.418630] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2695.670709] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2697.384223] Key type lgssc registered [ 2698.822882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2708.030754] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2708.122061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2723.809354] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2728.927258] LustreError: 76513:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 2755.743191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.868408] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2771.336166] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2780.607308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.730674] Lustre: *** cfs_fail_loc=706, val=0*** [ 2781.138239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.225535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2786.279374] LustreError: 76539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.313862] LustreError: 76539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2786.417071] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2790.372622] LustreError: 76538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.393366] LustreError: 76538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2795.493107] LustreError: 76539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.505162] LustreError: 76539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2796.978490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.454977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.146238] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2809.074363] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2813.786649] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2817.016993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2817.029462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.048742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2818.018946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.020299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2818.054487] Lustre: Skipped 1 previous similar message [ 2822.456918] Lustre: server umount lustre-OST0000 complete [ 2828.256696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.258774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2828.265739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.282674] Lustre: Skipped 1 previous similar message [ 2833.023609] Lustre: server umount lustre-MDT0000 complete [ 2833.385230] LustreError: 77490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.398708] LustreError: 77490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2837.391245] LustreError: 76520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220144 with bad export cookie 13392283482705041006 [ 2837.397705] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2837.412838] LustreError: 76520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2837.884324] Lustre: server umount lustre-MDT0001 complete [ 2850.072887] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2853.131885] Key type lgssc unregistered [ 2853.543787] LNet: 79775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2853.569451] LNetError: 79775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2853.598222] LNet: Removed LNI 192.168.202.136@tcp [ 2854.469165] Key type .llcrypt unregistered [ 2854.474160] Key type ._llcrypt unregistered [ 2884.429412] Key type ._llcrypt registered [ 2884.431804] Key type .llcrypt registered [ 2884.533241] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2886.161124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2886.188631] alg: No test for adler32 (adler32-zlib) [ 2887.423134] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 2887.743803] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2889.463545] Key type lgssc registered [ 2890.946867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2899.690423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2899.704085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.284652] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2914.505910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.064560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2920.410727] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2920.420114] LustreError: 81846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.434837] LustreError: 81846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2924.521142] LustreError: 81850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.555993] LustreError: 81850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2929.637708] LustreError: 81846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.656742] LustreError: 81846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2933.389211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.970063] LustreError: 81845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.010162] LustreError: 81845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2934.026044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.361631] LustreError: 81849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.383459] LustreError: 81849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2941.751883] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2950.630399] LustreError: 81850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.654176] LustreError: 81850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2951.795350] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2951.805221] Lustre: MGS: Client e342249f-bf5d-4fad-82a3-15ec1986a09a (at 0@lo) reconnecting [ 2951.840956] LustreError: 81860:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2952.084579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.183340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2956.746909] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2972.643469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2972.648128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.654831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.661048] Lustre: Skipped 1 previous similar message [ 2972.673138] Lustre: Skipped 1 previous similar message [ 2977.763591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.776711] Lustre: Skipped 1 previous similar message [ 2978.600373] Lustre: server umount lustre-OST0000 complete [ 2988.001247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.009094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.050727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2990.054119] Lustre: server umount lustre-MDT0000 complete [ 2993.122829] LustreError: 83357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.148594] LustreError: 83357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2995.904949] Lustre: server umount lustre-MDT0001 complete [ 2999.561877] Lustre: server umount MGS complete [ 3015.530734] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:25:20 (1781220320) [ 3052.788689] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3065.934372] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3110.489655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.830729] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3126.048672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.267692] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3126.305975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3126.382405] Lustre: lustre-MDT0001: new disk, initializing [ 3126.442534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3126.461844] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3126.470297] Lustre: 88944:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3126.480912] LustreError: 88944:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3131.435252] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3143.716923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3144.123477] Lustre: lustre-OST0000: new disk, initializing [ 3144.127909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3144.139777] Lustre: 89695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3144.232460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.952161] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3151.467376] Lustre: 89977:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c9a4451e x1867744746187904/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781220474 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3159.756256] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3168.817311] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.847854] Lustre: MGS: Client 81f282b0-bb75-4a83-86d2-3d9f4932bf7b (at 0@lo) reconnecting [ 3168.925484] Lustre: 90627:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3168.953152] LustreError: 88930:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3168.965689] LustreError: 88930:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3168.987110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3169.136336] Lustre: lustre-MDT0000: new disk, initializing [ 3169.377638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.415535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3169.448356] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3170.534457] Lustre: 90652:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c17f370b x1867744746195072/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781220493 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3174.737784] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3175.662547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3175.678465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3175.701514] Lustre: Skipped 1 previous similar message [ 3175.742598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3204.577807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.581951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.598185] Lustre: Skipped 1 previous similar message [ 3204.603196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.611429] Lustre: Skipped 1 previous similar message [ 3211.246924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3211.274344] Lustre: Skipped 3 previous similar messages [ 3216.351102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3216.592923] Lustre: server umount lustre-OST0000 complete [ 3221.477176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3221.480933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.493591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3221.518564] Lustre: Skipped 2 previous similar messages [ 3221.539561] Lustre: Skipped 3 previous similar messages [ 3226.982058] Lustre: server umount lustre-MDT0000 complete [ 3231.718991] LustreError: 88915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.737453] LustreError: 88915:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3238.479804] Lustre: server umount lustre-MDT0001 complete [ 3243.087638] Lustre: server umount MGS complete [ 3256.759789] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:29:21 (1781220561) [ 3278.076681] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3281.021185] Key type lgssc unregistered [ 3281.283187] LNet: 93124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3281.286840] LNetError: 93124:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3281.303755] LNet: Removed LNI 192.168.202.136@tcp [ 3282.202444] Key type .llcrypt unregistered [ 3282.204364] Key type ._llcrypt unregistered [ 3301.384360] Key type ._llcrypt registered [ 3301.389313] Key type .llcrypt registered [ 3301.610522] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3303.416981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.470659] alg: No test for adler32 (adler32-zlib) [ 3304.606627] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 3304.829660] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3306.503195] Key type lgssc registered [ 3307.752461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3317.092328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3317.125866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.901307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.193378] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3328.487977] LustreError: 94335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.530137] LustreError: 94335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3333.600703] LustreError: 94336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.650419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.171880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3339.557444] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3345.234718] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3349.358343] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3357.057563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.396781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.587035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3363.826435] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3370.917272] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3384.312386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.320459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3384.350195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3384.365708] LustreError: Skipped 1 previous similar message [ 3384.402799] Lustre: Skipped 1 previous similar message [ 3386.600628] Lustre: server umount lustre-OST0000 complete [ 3394.530031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.551776] Lustre: Skipped 1 previous similar message [ 3394.559848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3394.565526] Lustre: Skipped 1 previous similar message [ 3397.842728] Lustre: server umount lustre-MDT0000 complete [ 3399.654540] LustreError: 94336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.681324] LustreError: 94336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3403.168955] LustreError: 94315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220710 with bad export cookie 16908258868348619733 [ 3403.187796] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3403.699671] Lustre: server umount lustre-MDT0001 complete [ 3417.224217] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3421.127917] Key type lgssc unregistered [ 3421.642165] LNet: 97513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.654557] LNetError: 97513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.693079] LNet: Removed LNI 192.168.202.136@tcp [ 3422.659953] Key type .llcrypt unregistered [ 3422.664136] Key type ._llcrypt unregistered [ 3441.606951] Key type ._llcrypt registered [ 3441.610464] Key type .llcrypt registered [ 3441.729959] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3442.981597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3443.177617] alg: No test for adler32 (adler32-zlib) [ 3444.515638] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 3444.799376] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3446.559512] Key type lgssc registered [ 3448.011834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3455.112829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3455.118372] LustreError: 98579:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3455.151841] LustreError: 98584:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3455.156717] LustreError: 98579:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3455.167845] LustreError: 98579:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3455.176396] LustreError: 98579:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3455.187439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.832098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.604502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.109142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3471.465456] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3478.524331] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing lsmod [ 3482.686410] LustreError: 99752:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3482.749060] LustreError: 99759:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3482.754151] LustreError: 99752:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4945 [ 3482.759877] LustreError: 99752:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3482.771092] LustreError: 99752:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3482.784740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3483.263947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.691967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3486.701900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.726976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.741101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.743501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3488.783394] Lustre: Skipped 1 previous similar message [ 3488.788878] Lustre: Skipped 1 previous similar message [ 3490.861192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3490.881080] Lustre: Skipped 2 previous similar messages [ 3492.854650] Lustre: server umount lustre-MDT0000 complete [ 3495.910803] LustreError: 98623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.938046] LustreError: 98623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3496.896906] LustreError: 98595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220803 with bad export cookie 2894239687412550842 [ 3496.914369] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3497.056888] LustreError: 99053:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e0e1d272 x1867745330223488/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3497.408573] Lustre: server umount lustre-MDT0001 complete [ 3512.141845] Lustre: server umount lustre-OST0000 complete [ 3520.585254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.998088] LustreError: 100824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.087972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.118276] LustreError: 100825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.137327] LustreError: 100825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3526.454040] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3530.214412] LustreError: 100824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.330770] LustreError: 100825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.132393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.475920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3540.756800] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3546.433689] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3550.992750] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3558.322179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3558.661789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.183519] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3565.822121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3571.715345] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3581.397672] Lustre: server umount lustre-OST0000 complete [ 3586.528322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.548358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.551072] Lustre: Skipped 1 previous similar message [ 3591.649049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.661233] Lustre: Skipped 1 previous similar message [ 3597.791353] Lustre: 97984:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220888/real 1781220888] req@0000000006d7fabc x1867745330251008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781220904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3597.824220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.838869] Lustre: Skipped 1 previous similar message [ 3598.815180] Lustre: 97987:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220889/real 1781220889] req@00000000c9a4451e x1867745330251392/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781220905 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3600.352909] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3600.515741] Lustre: server umount lustre-MDT0000 complete [ 3602.914338] LustreError: 102977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.928136] LustreError: 102977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3603.935781] Lustre: 97987:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220893/real 1781220893] req@00000000970a90a4 x1867745330252800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781220909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3604.746213] LustreError: 100806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220911 with bad export cookie 2894239687412551542 [ 3604.748017] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.757383] LustreError: 100806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3605.234604] Lustre: server umount lustre-MDT0001 complete [ 3618.154860] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3621.397698] Key type lgssc unregistered [ 3621.663456] LNet: 104004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.672489] LNetError: 104004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.685366] LNet: Removed LNI 192.168.202.136@tcp [ 3622.723938] Key type .llcrypt unregistered [ 3622.727064] Key type ._llcrypt unregistered [ 3651.098017] Key type ._llcrypt registered [ 3651.106025] Key type .llcrypt registered [ 3651.302209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3652.613891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3652.783434] alg: No test for adler32 (adler32-zlib) [ 3653.883299] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 3654.131563] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3655.847201] Key type lgssc registered [ 3657.452868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3666.956037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3666.975898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.593432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.014863] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3678.190772] LustreError: 105405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.224830] LustreError: 105405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3683.303012] LustreError: 105404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.102108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.692814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3691.730978] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3699.134845] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3704.244542] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3713.045887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3713.484804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.568072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3720.360883] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3726.709112] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3736.115142] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3740.798634] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3745.255553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3745.266489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.295095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.333198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.336584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.338643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.393705] Lustre: server umount lustre-OST0000 complete [ 3758.572726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.586256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.593134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3758.595790] Lustre: Skipped 1 previous similar message [ 3758.628798] Lustre: Skipped 2 previous similar messages [ 3761.908823] Lustre: server umount lustre-MDT0000 complete [ 3763.682826] LustreError: 107608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.692059] LustreError: 107608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3766.911939] LustreError: 107596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221073 with bad export cookie 6780232042939515036 [ 3766.917560] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3766.926727] LustreError: 107596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3767.734592] Lustre: server umount lustre-MDT0001 complete [ 3782.656980] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3786.506219] Key type lgssc unregistered [ 3786.906862] LNet: 108688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.921425] LNetError: 108688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3786.994156] LNet: Removed LNI 192.168.202.136@tcp [ 3788.456326] Key type .llcrypt unregistered [ 3788.459396] Key type ._llcrypt unregistered [ 3813.372767] Key type ._llcrypt registered [ 3813.376741] Key type .llcrypt registered [ 3813.585521] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3814.831096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3814.922093] alg: No test for adler32 (adler32-zlib) [ 3816.262549] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 3816.686309] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3818.383233] Key type lgssc registered [ 3819.770809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3829.151922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3829.165968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.068194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.480694] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3840.487885] LustreError: 109894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3840.514827] LustreError: 109894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3845.602116] LustreError: 109895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.713673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.151492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3852.633791] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3859.433119] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3864.059569] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3871.811306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3872.294735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.420525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3880.077362] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3886.387910] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3900.901427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3900.919834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.942674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.924503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.925763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.936196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.974473] Lustre: Skipped 1 previous similar message [ 3905.398947] Lustre: server umount lustre-OST0000 complete [ 3912.163663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.171404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3912.193213] Lustre: Skipped 1 previous similar message [ 3912.210622] Lustre: Skipped 1 previous similar message [ 3916.524231] Lustre: server umount lustre-MDT0000 complete [ 3917.285460] LustreError: 109891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.305249] LustreError: 109891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3920.523898] LustreError: 109875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221227 with bad export cookie 3397502428727096269 [ 3920.528603] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.532340] LustreError: 109875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3920.805311] Lustre: server umount lustre-MDT0001 complete [ 3933.573157] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3937.189476] Key type lgssc unregistered [ 3937.641646] LNet: 113095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3937.645367] LNetError: 113095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3938.662540] LNet: Removed LNI 192.168.202.136@tcp [ 3939.783180] Key type .llcrypt unregistered [ 3939.786828] Key type ._llcrypt unregistered [ 3972.853206] Key type ._llcrypt registered [ 3972.855675] Key type .llcrypt registered [ 3973.048337] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3975.774757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3975.907326] alg: No test for adler32 (adler32-zlib) [ 3977.059425] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 3977.430849] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3979.183182] Key type lgssc registered [ 3980.772642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3991.082155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3991.123156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.659817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.336681] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4002.787577] LustreError: 114496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4002.798563] LustreError: 114496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4005.791921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.198479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4010.497995] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4017.523884] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4021.377661] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4029.130829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4029.426776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.025704] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4037.630367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4044.075961] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4051.452419] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4056.385552] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4056.397298] Lustre: Skipped 2 previous similar messages [ 4057.164251] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4057.188508] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4061.694207] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4061.701228] Lustre: Skipped 2 previous similar messages [ 4062.425098] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4062.434364] Lustre: Skipped 1 previous similar message [ 4071.410842] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4071.419718] Lustre: Skipped 2 previous similar messages [ 4072.127914] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.36@tcp 192.168.0.[2,10]@tcp [ 4079.816644] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.36/24@tcp in log lustre-MDT0000 [ 4079.836692] Lustre: Skipped 2 previous similar messages [ 4085.990303] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.36/24@tcp [ 4086.003690] Lustre: Skipped 1 previous similar message [ 4096.995769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4097.002586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.030183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.044497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.048857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4103.187795] Lustre: server umount lustre-OST0000 complete [ 4108.768383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4108.786110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4108.806217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4108.813141] Lustre: Skipped 1 previous similar message [ 4114.401069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4114.418293] Lustre: Skipped 3 previous similar messages [ 4115.017272] Lustre: server umount lustre-MDT0000 complete [ 4119.367870] LustreError: 114478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221426 with bad export cookie 12390744929316290283 [ 4119.379353] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.384343] LustreError: 114478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4119.758396] Lustre: server umount lustre-MDT0001 complete [ 4132.069709] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4135.431356] Key type lgssc unregistered [ 4135.763922] LNet: 118846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4135.776759] LNetError: 118846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4135.796395] LNet: Removed LNI 192.168.202.136@tcp [ 4136.992529] Key type .llcrypt unregistered [ 4136.995081] Key type ._llcrypt unregistered [ 4174.247747] Key type ._llcrypt registered [ 4174.249817] Key type .llcrypt registered [ 4174.369476] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4175.207913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4175.248069] alg: No test for adler32 (adler32-zlib) [ 4176.525516] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 4176.702139] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4178.367183] Key type lgssc registered [ 4179.797987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4189.193860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4189.222783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.737302] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4190.801205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4190.801594] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.36/24@tcp [ 4195.179626] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4200.942143] LustreError: 120392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.967847] LustreError: 120392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4205.789486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.187497] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4206.253336] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.36/24@tcp [ 4206.309799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4212.414292] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4222.206858] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4227.714470] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4237.619062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.056805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.770995] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4253.701677] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4260.753041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4276.195097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.198038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.199075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4276.216918] Lustre: Skipped 1 previous similar message [ 4276.246849] Lustre: Skipped 1 previous similar message [ 4280.685661] Lustre: server umount lustre-OST0000 complete [ 4286.345581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.355594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.363414] Lustre: Skipped 1 previous similar message [ 4291.552638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4291.573135] Lustre: Skipped 1 previous similar message [ 4293.110602] Lustre: server umount lustre-MDT0000 complete [ 4296.683964] LustreError: 122550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.724965] LustreError: 122550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4298.662839] LustreError: 120372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221605 with bad export cookie 15479514745455262897 [ 4298.681102] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4299.377728] Lustre: server umount lustre-MDT0001 complete [ 4314.246595] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4318.262560] Key type lgssc unregistered [ 4318.686996] LNet: 123652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4318.700916] LNetError: 123652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4318.726427] LNet: Removed LNI 192.168.202.136@tcp [ 4319.787873] Key type .llcrypt unregistered [ 4319.790303] Key type ._llcrypt unregistered [ 4353.143818] Key type ._llcrypt registered [ 4353.149852] Key type .llcrypt registered [ 4353.258414] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4370.677859] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4372.116505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.156890] alg: No test for adler32 (adler32-zlib) [ 4373.216506] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 4373.440113] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4375.154581] Key type lgssc registered [ 4376.227871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4419.204835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4419.236714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.647784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4420.740471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4420.829157] Lustre: lustre-MDT0000: new disk, initializing [ 4420.971032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.996550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4425.894270] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4438.010346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.184225] Lustre: 127022:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4438.264832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4438.277710] Lustre: Skipped 1 previous similar message [ 4438.414800] Lustre: lustre-MDT0001: new disk, initializing [ 4438.587536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4438.657227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4438.678575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4443.789206] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4452.909184] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4458.221581] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4466.822427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.284691] Lustre: lustre-OST0000: new disk, initializing [ 4467.290838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4467.304697] Lustre: 128763:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4467.497967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.247628] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4476.512783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4476.533025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4476.686285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4483.742193] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4489.086990] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4489.276972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4493.576859] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4493.737546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4497.377724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.378855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4497.398392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4497.410728] Lustre: Skipped 1 previous similar message [ 4502.521471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4502.525361] Lustre: Skipped 1 previous similar message [ 4507.617556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4507.628622] Lustre: Skipped 1 previous similar message [ 4510.175671] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4510.361600] Lustre: server umount lustre-OST0000 complete [ 4517.857957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.864670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4517.890112] Lustre: Skipped 2 previous similar messages [ 4522.977156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.989762] Lustre: Skipped 2 previous similar messages [ 4530.143833] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4530.584558] Lustre: server umount lustre-MDT0000 complete [ 4534.506790] LustreError: 127014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221841 with bad export cookie 9532379384908523412 [ 4534.511693] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.516112] LustreError: 127014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4534.747196] Lustre: server umount lustre-MDT0001 complete [ 4543.775200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.182209] LustreError: 130839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4544.203474] LustreError: 130839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4544.243762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.717312] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4549.613911] LustreError: 130840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.725845] LustreError: 130839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.374139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.729078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.712489] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4567.747765] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4572.484857] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4581.406122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4581.784801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.460901] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4598.088607] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4603.184615] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4603.396178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4608.136531] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4608.296288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4618.645135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.820358] Lustre: lustre-OST0001: new disk, initializing [ 4618.824397] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4618.829792] Lustre: 133839:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4618.945979] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4626.266971] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4627.089393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4627.111487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4627.244731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4634.227449] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4634.392759] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4639.482815] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4639.736607] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4645.147143] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942f47505000.ost_server_uuid 50 [ 4647.253397] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff942f47505000.ost_server_uuid in IDLE state after 0 sec [ 4660.668431] Lustre: server umount lustre-OST0001 complete [ 4663.265135] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4663.273222] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4663.299601] LustreError: 132434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.309582] Lustre: Skipped 1 previous similar message [ 4663.328719] LustreError: 132434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4666.854188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4666.871146] LustreError: Skipped 1 previous similar message [ 4666.880930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4666.935581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4666.952199] Lustre: Skipped 2 previous similar messages [ 4668.391332] LustreError: 132435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.407023] LustreError: 132435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4671.419510] Lustre: server umount lustre-OST0000 complete [ 4675.552448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4675.560914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.575072] Lustre: Skipped 1 previous similar message [ 4681.834342] Lustre: server umount lustre-MDT0000 complete [ 4683.754878] LustreError: 130834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.772738] LustreError: 130834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4685.456931] LustreError: 130820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221992 with bad export cookie 9532379384908524294 [ 4685.460828] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4685.477950] LustreError: 130820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4685.906732] Lustre: server umount lustre-MDT0001 complete [ 4706.292202] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4709.615760] Key type lgssc unregistered [ 4709.880607] LNet: 136676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4709.886024] LNetError: 136676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4709.920533] LNet: Removed LNI 192.168.202.136@tcp [ 4710.942534] Key type .llcrypt unregistered [ 4710.944625] Key type ._llcrypt unregistered [ 4767.690808] Key type ._llcrypt registered [ 4767.701690] Key type .llcrypt registered [ 4767.877851] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4786.175622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4787.597757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4787.655488] alg: No test for adler32 (adler32-zlib) [ 4788.932434] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 4789.272359] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4790.983279] Key type lgssc registered [ 4792.368948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4839.463635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4839.523055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.872877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4840.912994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4840.993209] Lustre: lustre-MDT0000: new disk, initializing [ 4841.065099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.083829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4845.997778] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4858.362937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.618466] Lustre: 140801:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4858.759735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4858.774514] Lustre: Skipped 1 previous similar message [ 4858.881577] Lustre: lustre-MDT0001: new disk, initializing [ 4859.019530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4859.069247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4859.095358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4863.595316] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4872.811342] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4877.816810] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.189292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.533968] Lustre: lustre-OST0000: new disk, initializing [ 4886.540628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4886.552962] Lustre: 142544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4886.771268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.805383] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4895.347860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4895.383325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4895.575881] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4905.038543] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4909.705913] Lustre: Failing over lustre-OST0000 [ 4911.080667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.092098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4911.105924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.120374] Lustre: Skipped 1 previous similar message [ 4912.125057] Lustre: server umount lustre-OST0000 complete [ 4933.069904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.606042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.622902] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4938.706256] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4941.937718] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4947.498850] Lustre: Failing over lustre-MDT0000 [ 4948.644486] Lustre: server umount lustre-MDT0000 complete [ 4951.012602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4951.020361] LustreError: Skipped 1 previous similar message [ 4951.034258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.057618] Lustre: Skipped 1 previous similar message [ 4952.052770] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.098267] Lustre: Skipped 1 previous similar message [ 4954.057375] LustreError: 141855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4954.096369] LustreError: 141855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4958.188810] LustreError: 140812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.233034] LustreError: 140812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4963.301551] LustreError: 140808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.328026] LustreError: 140808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4965.424791] LustreError: 140809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4965.445952] LustreError: 140809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4967.392138] Lustre: 139023:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781222258/real 1781222258] req@0000000026fddad7 x1867746740582656/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781222274 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4967.413522] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4970.437041] LustreError: 141855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.479720] LustreError: 141855:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4971.140466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.636734] LustreError: 139020:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000076872ab5 x1867746740586880/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4977.979061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4978.107319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.158408] Lustre: lustre-OST0000: Recovery over after 0:40, of 3 clients 3 recovered and 0 were evicted. [ 4978.160280] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4978.265057] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4979.657747] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4983.277045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4983.286703] Lustre: Skipped 1 previous similar message [ 4983.318568] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4983.380809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4983.521592] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4998.627695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4998.641050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4998.649584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4998.664242] Lustre: Skipped 1 previous similar message [ 5000.550221] Lustre: server umount lustre-OST0000 complete [ 5008.877701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.885469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5008.885490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5008.899861] Lustre: Skipped 2 previous similar messages [ 5008.928481] Lustre: Skipped 1 previous similar message [ 5012.037867] Lustre: server umount lustre-MDT0000 complete [ 5013.998691] LustreError: 140809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.033246] LustreError: 140809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5017.127649] LustreError: 141589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222324 with bad export cookie 14419937460554922918 [ 5017.134078] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.142138] LustreError: 141589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5017.803991] Lustre: server umount lustre-MDT0001 complete [ 5030.951312] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5034.383019] Key type lgssc unregistered [ 5034.682064] LNet: 146040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5034.696763] LNetError: 146040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5034.710818] LNet: Removed LNI 192.168.202.136@tcp [ 5035.657959] Key type .llcrypt unregistered [ 5035.660286] Key type ._llcrypt unregistered [ 5076.474137] Key type ._llcrypt registered [ 5076.479170] Key type .llcrypt registered [ 5076.656676] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5078.055190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5078.199934] alg: No test for adler32 (adler32-zlib) [ 5079.368731] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 5079.752724] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5081.495279] Key type lgssc registered [ 5082.754614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5091.633671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5091.649137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.959435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.124489] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5103.074151] LustreError: 147545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.121910] LustreError: 147545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5108.194242] LustreError: 147546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5110.422960] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5113.315778] LustreError: 147545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.122150] LustreError: 147574:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5115.132792] LustreError: 147574:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5115.140357] LustreError: 147574:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 5115.866433] Lustre: server umount lustre-MDT0000 complete [ 5139.122718] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5151.343697] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5197.707342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.842890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5197.856234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5197.907385] Lustre: lustre-MDT0000: new disk, initializing [ 5197.944779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.953511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5203.489704] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5216.204176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.349637] Lustre: 150744:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5216.447766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5216.458505] Lustre: Skipped 1 previous similar message [ 5216.616038] Lustre: lustre-MDT0001: new disk, initializing [ 5216.813760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5216.876786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5216.890584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5222.334541] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5230.726649] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5235.152461] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5243.534244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.759414] Lustre: lustre-OST0000: new disk, initializing [ 5243.763969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5243.770987] Lustre: 152487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5243.887809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.458287] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5253.216288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5253.235251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5253.369181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5260.909929] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5265.984474] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5268.127090] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:02:52 (1781222572) [ 5559.781877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5559.785654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.804922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.810562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.818261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.674621] Lustre: server umount lustre-MDT0000 complete [ 5565.922436] LustreError: 151890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.945487] LustreError: 151890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5568.024729] LustreError: 150737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222874 with bad export cookie 7148165486247050713 [ 5568.028220] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.038083] LustreError: 150737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5571.040899] LustreError: 150753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.044127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.094130] Lustre: Skipped 1 previous similar message [ 5571.115427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5571.124896] Lustre: Skipped 2 previous similar messages [ 5574.813387] Lustre: server umount lustre-MDT0001 complete [ 5586.298538] Lustre: server umount lustre-OST0000 complete [ 5598.808752] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5612.777413] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5654.519612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.842431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5654.869572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5654.989201] Lustre: lustre-MDT0000: new disk, initializing [ 5655.061986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.084314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5659.697754] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5673.031455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.273750] Lustre: 157029:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5673.392922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5673.396758] Lustre: Skipped 1 previous similar message [ 5673.554463] Lustre: lustre-MDT0001: new disk, initializing [ 5673.698975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5673.777371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5673.793960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5679.290542] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5689.034149] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5694.430317] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5705.862719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5706.363621] Lustre: lustre-OST0000: new disk, initializing [ 5706.387554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5706.420575] Lustre: 158774:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5706.628906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.585367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5713.618252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5713.817262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5715.767762] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5723.487641] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5729.858947] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5730.148204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5735.459132] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5735.773172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5739.497443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5739.498201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.510623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5739.510630] Lustre: Skipped 1 previous similar message [ 5739.545452] Lustre: Skipped 1 previous similar message [ 5744.423168] Lustre: server umount lustre-OST0000 complete [ 5749.741291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.750107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5749.764822] Lustre: Skipped 1 previous similar message [ 5749.788776] Lustre: Skipped 2 previous similar messages [ 5754.851389] LustreError: 158160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.871947] LustreError: 158160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5754.888797] Lustre: server umount lustre-MDT0000 complete [ 5759.439262] LustreError: 157021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223066 with bad export cookie 7148165486247209543 [ 5759.447035] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.454455] LustreError: 157021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5759.898936] Lustre: server umount lustre-MDT0001 complete [ 5769.649083] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:11:14 (1781223074) [ 5787.161193] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5800.002678] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5850.664652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.031787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5851.072787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5851.176276] Lustre: lustre-MDT0000: new disk, initializing [ 5851.266617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.284288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5856.264167] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5868.839727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.052334] Lustre: 163353:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5869.414067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5874.677083] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5883.159613] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5887.897310] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5895.916836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.372118] Lustre: lustre-OST0000: new disk, initializing [ 5896.374530] Lustre: Skipped 1 previous similar message [ 5896.378907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5896.384169] Lustre: Skipped 2 previous similar messages [ 5896.389603] Lustre: 165103:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5902.279924] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5905.463165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5905.475208] Lustre: Skipped 1 previous similar message [ 5905.483841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5905.598170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5911.865389] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5921.255268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.259598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5921.303676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.306632] LustreError: Skipped 1 previous similar message [ 5921.336865] Lustre: Skipped 1 previous similar message [ 5926.400362] LustreError: 165082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.422974] LustreError: 165082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5926.747265] Lustre: server umount lustre-OST0000 complete [ 5930.976634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5936.919114] Lustre: server umount lustre-MDT0000 complete [ 5942.681455] LustreError: 165102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223249 with bad export cookie 7148165486247210425 [ 5942.702390] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5942.707090] LustreError: 165102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5943.571081] Lustre: server umount lustre-MDT0001 complete [ 5957.753634] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:14:21 (1781223261) [ 5978.037583] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5990.251910] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6040.106692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.324779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6040.349526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6040.435987] Lustre: lustre-MDT0000: new disk, initializing [ 6040.538318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.544520] Lustre: Skipped 2 previous similar messages [ 6040.563511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6047.432629] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6060.556062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.742626] Lustre: 169284:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6061.161707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6066.769744] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6076.370475] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6081.076371] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6090.483514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.947984] Lustre: 171035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6096.536466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6096.687933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6099.639442] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6108.096623] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6122.471855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.477598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.499158] Lustre: Skipped 3 previous similar messages [ 6122.511382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.528041] Lustre: Skipped 4 previous similar messages [ 6124.311528] Lustre: server umount lustre-OST0000 complete [ 6132.713303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6136.942255] Lustre: server umount lustre-MDT0000 complete [ 6137.824845] LustreError: 170626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.850716] LustreError: 170626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6141.333499] LustreError: 170066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223448 with bad export cookie 7148165486247211510 [ 6141.341936] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.360486] LustreError: 170066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6141.745340] Lustre: server umount lustre-MDT0001 complete [ 6155.901587] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6158.967536] Key type lgssc unregistered [ 6159.267884] LNet: 172763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6159.275566] LNetError: 172763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6160.311462] LNet: Removed LNI 192.168.202.136@tcp [ 6161.311123] Key type .llcrypt unregistered [ 6161.313791] Key type ._llcrypt unregistered [ 6198.064757] Key type ._llcrypt registered [ 6198.066461] Key type .llcrypt registered [ 6198.299174] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6200.865408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6200.995073] alg: No test for adler32 (adler32-zlib) [ 6202.715739] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 6203.508310] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6205.303877] Key type lgssc registered [ 6207.953106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.479092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.500506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.308321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.169610] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6233.573569] LustreError: 174165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.602723] LustreError: 174165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6238.690182] LustreError: 174166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.640666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.255466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6247.983377] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6256.659495] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6262.920834] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6271.217759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.518370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.926636] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6288.259634] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6294.098231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6312.428820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.435528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.443728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.471409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.478787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6314.495240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.145911] Lustre: server umount lustre-OST0000 complete [ 6323.169969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6323.181100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.210254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6323.223556] Lustre: Skipped 1 previous similar message [ 6328.616218] Lustre: server umount lustre-MDT0000 complete [ 6329.831254] LustreError: 174161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.849182] LustreError: 174161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6333.517846] LustreError: 174146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223640 with bad export cookie 13838934983860272460 [ 6333.546221] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6333.892822] Lustre: server umount lustre-MDT0001 complete [ 6348.202206] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6351.126298] Key type lgssc unregistered [ 6351.491409] LNet: 177373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6351.513542] LNetError: 177373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6351.536415] LNet: Removed LNI 192.168.202.136@tcp [ 6352.394291] Key type .llcrypt unregistered [ 6352.396846] Key type ._llcrypt unregistered [ 6382.795348] Key type ._llcrypt registered [ 6382.799096] Key type .llcrypt registered [ 6382.951417] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6384.457427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6384.688509] alg: No test for adler32 (adler32-zlib) [ 6385.808837] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 6386.028800] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6387.751856] Key type lgssc registered [ 6389.152696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6399.762685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6399.785439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6401.422497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6406.871126] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6411.746086] LustreError: 178773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.768328] LustreError: 178773:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6416.879751] LustreError: 178774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.571459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.152746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6425.283428] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6434.804914] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6440.199569] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6449.102357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.782124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.362452] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6466.290926] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6471.283241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6479.842176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6479.854553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6479.883451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6481.889046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6481.895121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6481.915431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.962381] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6483.982175] Lustre: Skipped 1 previous similar message [ 6485.971048] Lustre: server umount lustre-OST0000 complete [ 6493.749376] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff942f46b69800.ost_server_uuid 50 [ 6496.831108] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6516.709759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.712437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6516.749417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6516.762417] Lustre: Skipped 1 previous similar message [ 6519.046753] Lustre: server umount lustre-MDT0000 complete [ 6521.826096] LustreError: 178774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.856968] LustreError: 178774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6523.983916] LustreError: 178754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223830 with bad export cookie 16273004540882105582 [ 6523.984617] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6524.021317] LustreError: 178754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6524.784447] Lustre: server umount lustre-MDT0001 complete [ 6535.420400] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:23:59 (1781223839) [ 6542.919520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.719860] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.911858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6548.965368] LustreError: 182277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.848561] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6559.202871] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.230119] LustreError: 182276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6563.603315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.340370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6569.913439] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6577.427907] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6583.569647] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6591.988758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6592.386033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.722131] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6606.592576] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6613.965412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6615.277458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6615.516336] Lustre: lustre-OST0001: new disk, initializing [ 6615.526703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6615.535146] Lustre: 184783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6615.637264] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6622.793179] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6624.876973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6624.904360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6625.059393] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6633.143737] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6639.587590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6639.603343] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6639.603768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.628420] Lustre: Skipped 1 previous similar message [ 6643.436599] Lustre: server umount lustre-OST0000 complete [ 6644.679513] LustreError: 183873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.753686] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6652.064915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6657.861276] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6658.172561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6676.448738] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.467937] LustreError: Skipped 1 previous similar message [ 6676.476867] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6676.488373] Lustre: Skipped 1 previous similar message [ 6676.501423] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6676.505549] Lustre: Skipped 4 previous similar messages [ 6679.282981] Lustre: server umount lustre-OST0001 complete [ 6686.690663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.700991] Lustre: Skipped 1 previous similar message [ 6690.261108] Lustre: server umount lustre-MDT0000 complete [ 6691.811902] LustreError: 182988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.837692] LustreError: 182988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 6694.577810] LustreError: 182256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224001 with bad export cookie 16273004540882106653 [ 6694.580961] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.587281] LustreError: 182256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6695.535332] Lustre: server umount lustre-MDT0001 complete [ 6722.927812] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:27:07 (1781224027) [ 6730.711614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.038888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6731.078964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6731.350244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6736.776332] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6747.447515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.644992] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6747.699511] Lustre: 188074:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6753.876255] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6760.662408] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6765.159240] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6772.994030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.200685] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6781.798226] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6783.163566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6788.366640] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6797.518424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.633907] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6798.011692] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6798.014925] Lustre: Skipped 2 previous similar messages [ 6806.110458] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6807.236538] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6812.793481] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6817.776113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.792054] Lustre: Skipped 1 previous similar message [ 6817.797315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6817.811204] Lustre: Skipped 3 previous similar messages [ 6822.102632] Lustre: server umount lustre-OST0000 complete [ 6822.928736] LustreError: 189703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.954093] LustreError: 189703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6842.454818] Lustre: server umount lustre-OST0001 complete [ 6853.601566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.622238] Lustre: Skipped 4 previous similar messages [ 6858.905645] Lustre: server umount lustre-MDT0000 complete [ 6859.751661] Lustre: 178086:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781224150/real 1781224150] req@00000000312a77da x1867748414227968/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781224166 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6859.811781] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.824836] Lustre: Skipped 3 previous similar messages [ 6864.014297] LustreError: 188067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224170 with bad export cookie 16273004540882108060 [ 6864.018735] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6864.033379] LustreError: 188067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6864.613472] Lustre: server umount lustre-MDT0001 complete [ 6897.128617] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:30:01 (1781224201) [ 6917.383761] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 6931.055609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6981.455121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.712674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6981.748870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6981.858407] Lustre: lustre-MDT0000: new disk, initializing [ 6981.986732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6982.015898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6988.262593] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7003.288201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.487650] Lustre: 195689:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7003.581140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7003.588474] Lustre: Skipped 1 previous similar message [ 7003.772476] Lustre: lustre-MDT0001: new disk, initializing [ 7004.011110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7004.034328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7010.334840] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7019.991563] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7024.882487] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7034.607705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.962642] Lustre: lustre-OST0000: new disk, initializing [ 7034.967803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7034.977777] Lustre: 197436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7042.139024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7042.160525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7042.327917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7043.883195] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7053.944564] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7060.769457] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7060.995708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7065.847586] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7066.162462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7073.252540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7073.255857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.264670] LustreError: Skipped 2 previous similar messages [ 7073.274543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7073.311121] Lustre: Skipped 1 previous similar message [ 7073.334880] Lustre: Skipped 3 previous similar messages [ 7074.511787] Lustre: server umount lustre-OST0000 complete [ 7093.216531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7093.416784] Lustre: server umount lustre-MDT0000 complete [ 7093.728569] LustreError: 195699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.759534] LustreError: 195699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7097.530440] LustreError: 198831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224404 with bad export cookie 16273004540882109845 [ 7097.536758] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7097.560677] LustreError: 198831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7097.987367] Lustre: server umount lustre-MDT0001 complete [ 7107.692248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7113.948226] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7124.658510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.502443] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7137.766066] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7142.393944] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7150.249661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7157.535708] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7164.408401] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7171.078441] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7171.283414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7176.997854] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7177.211523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7184.864334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7184.900522] LustreError: Skipped 2 previous similar messages [ 7186.336186] Lustre: server umount lustre-OST0000 complete [ 7193.225396] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7197.885958] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7229.810043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.222179] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7244.733488] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7246.974396] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7249.773728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7249.805366] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7249.807664] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7249.863842] Lustre: Skipped 1 previous similar message [ 7255.021122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.022871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.040282] LustreError: Skipped 1 previous similar message [ 7255.057450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.089656] Lustre: Skipped 5 previous similar messages [ 7255.136375] Lustre: Skipped 8 previous similar messages [ 7258.515236] Lustre: server umount lustre-OST0000 complete [ 7270.886478] Lustre: server umount lustre-MDT0000 complete [ 7275.309018] LustreError: 202245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224582 with bad export cookie 16273004540882110727 [ 7275.329319] LustreError: 202245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7275.329766] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7275.667193] Lustre: server umount lustre-MDT0001 complete [ 7285.898645] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:36:30 (1781224590) [ 7294.627251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7295.243601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7295.253735] Lustre: Skipped 6 previous similar messages [ 7301.059059] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7313.417681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7319.864675] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7328.326323] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7332.741089] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7340.401765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.950974] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7355.217706] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7360.274935] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7360.448786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7365.483858] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7365.828468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7374.008958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7374.378258] Lustre: lustre-OST0001: new disk, initializing [ 7374.387432] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7374.410137] Lustre: 207955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7383.492320] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7384.737676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7384.758224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7384.960575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7394.455465] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7400.605973] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7400.970299] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7406.965632] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7407.221121] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7410.664170] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7415.797423] LustreError: 206554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7415.815301] LustreError: 206554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7416.077937] Lustre: server umount lustre-OST0001 complete [ 7423.633424] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7423.899528] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7429.318628] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7429.655527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7462.533745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.021814] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7468.046868] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7468.063331] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7470.767065] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7479.196294] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7481.672638] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7490.681710] Lustre: server umount lustre-OST0001 complete [ 7513.056814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.076959] Lustre: Skipped 6 previous similar messages [ 7513.091216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7513.106525] Lustre: Skipped 8 previous similar messages [ 7516.968169] Lustre: server umount lustre-OST0000 complete [ 7528.521592] Lustre: server umount lustre-MDT0000 complete [ 7533.089420] LustreError: 204938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224839 with bad export cookie 16273004540882111693 [ 7533.099815] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7533.108715] LustreError: 204938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7539.701073] Lustre: server umount lustre-MDT0001 complete [ 7573.341835] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:41:17 (1781224877) [ 7581.925463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.349987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7582.431773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7587.942929] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7599.328620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7599.533821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7599.580325] Lustre: 212909:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7605.547547] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7613.293104] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7619.761141] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7628.996873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7629.234587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7638.330508] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7646.508629] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7661.192128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7661.482217] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7669.507402] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7678.583494] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7680.676320] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7686.619485] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7687.026898] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7693.132573] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7693.518426] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7698.599811] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942f510b5000.ost_server_uuid 50 [ 7700.898945] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff942f510b5000.ost_server_uuid in IDLE state after 0 sec [ 7702.392513] Lustre: Permanently deactivating lustre-OST0001 [ 7702.459845] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7707.482855] Lustre: Permanently reactivating lustre-OST0001 [ 7707.531421] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7707.540285] Lustre: Skipped 2 previous similar messages [ 7718.380157] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7718.392279] LustreError: Skipped 5 previous similar messages [ 7720.376154] Lustre: server umount lustre-OST0001 complete [ 7739.359520] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7739.623705] Lustre: server umount lustre-OST0000 complete [ 7750.666369] Lustre: server umount lustre-MDT0000 complete [ 7755.291043] LustreError: 212900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225062 with bad export cookie 16273004540882112981 [ 7755.300386] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7755.320563] LustreError: 212900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7755.972709] Lustre: server umount lustre-MDT0001 complete [ 7788.343678] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:44:52 (1781225092) [ 7797.683947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7798.159564] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7798.203911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7798.222489] Lustre: Skipped 2 previous similar messages [ 7803.701277] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7814.960266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.099095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7815.143690] Lustre: 219026:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7815.647116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7815.671238] Lustre: Skipped 9 previous similar messages [ 7821.522363] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7829.446974] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7835.061776] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7843.617691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.913984] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7843.956432] Lustre: Permanently deactivating lustre-OST0000 [ 7843.970479] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7850.584107] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7852.917543] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7861.027523] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7871.724075] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.829297] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7880.384500] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7882.152530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7888.793044] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7891.564633] Lustre: Permanently reactivating lustre-OST0000 [ 7891.597740] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7891.608690] Lustre: Skipped 2 previous similar messages [ 7899.590067] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 49 seconds. I think it's dead, and I am evicting it. exp 00000000bd71eaab, cur 1781225206 deadline 1781225187 last 1781225157 [ 7900.863819] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7900.878562] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7900.893895] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7900.907861] Lustre: Skipped 1 previous similar message [ 7900.971100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7912.701123] Lustre: server umount lustre-OST0001 complete [ 7928.322239] LustreError: 220652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.341716] LustreError: 220652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 54 previous similar messages [ 7932.383254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7932.549390] Lustre: server umount lustre-OST0000 complete [ 7943.757145] Lustre: server umount lustre-MDT0000 complete [ 7948.867331] LustreError: 219019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225255 with bad export cookie 16273004540882114647 [ 7948.887371] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7949.600620] Lustre: server umount lustre-MDT0001 complete [ 7963.471886] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7967.246939] Key type lgssc unregistered [ 7967.587808] LNet: 223446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7967.593857] LNetError: 223446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7967.617974] LNet: Removed LNI 192.168.202.136@tcp [ 7968.818392] Key type .llcrypt unregistered [ 7968.824183] Key type ._llcrypt unregistered [ 7998.209221] Key type ._llcrypt registered [ 7998.213362] Key type .llcrypt registered [ 7998.368231] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7999.805103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7999.841472] alg: No test for adler32 (adler32-zlib) [ 8000.906665] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 8001.094637] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8002.775578] Key type lgssc registered [ 8004.041792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8018.062458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8018.107305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.769391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.838815] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8029.163343] LustreError: 224899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8029.185251] LustreError: 224899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8034.297312] LustreError: 224898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.950406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.159193] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8036.203286] Lustre: 224888:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8036.215299] Lustre: Permanently deactivating lustre-MDT0001 [ 8036.242451] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8036.888823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8042.784640] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8043.566161] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8050.328995] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8055.729936] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8064.349596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8064.486361] LustreError: 226517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8064.695233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.132556] LustreError: 226516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.176586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8070.182365] LustreError: 226516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8071.590826] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8075.242991] LustreError: 226797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8078.792488] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8086.504835] LustreError: 226797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8086.538550] LustreError: 226797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8087.788135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.248182] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8093.699111] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8093.724048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8095.329549] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8103.334178] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8107.059762] Lustre: Permanently reactivating lustre-MDT0001 [ 8107.100078] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8107.105122] Lustre: Skipped 2 previous similar messages [ 8110.202068] Lustre: lustre-MDT0001: Client c6af62fa-d659-4993-81c8-b8f52168a07d (at 192.168.202.36@tcp) reconnecting [ 8115.456844] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8115.478349] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8115.497878] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8115.530920] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8115.900101] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8118.443750] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8127.307118] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8127.641355] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8133.715562] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8134.162575] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8137.488392] Lustre: Permanently deactivating lustre-MDT0001 [ 8137.552218] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8137.563928] Lustre: Skipped 2 previous similar messages [ 8144.523809] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8154.592924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.614518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.646877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.160811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.163590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.863747] Lustre: server umount lustre-MDT0000 complete [ 8162.275923] LustreError: 224893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8162.311777] LustreError: 224893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8166.231101] LustreError: 225620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225473 with bad export cookie 2683324032929263811 [ 8166.246609] LustreError: 225620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8166.249722] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8166.675527] Lustre: server umount lustre-MDT0001 complete [ 8173.453780] Lustre: server umount lustre-OST0000 complete [ 8189.270503] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:51:33 (1781225493) [ 8205.896801] Lustre: server umount lustre-OST0001 complete [ 8213.422912] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8228.645389] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8279.393359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.851585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8279.859446] Lustre: Skipped 2 previous similar messages [ 8279.941127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8280.079809] Lustre: lustre-MDT0000: new disk, initializing [ 8280.254315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8280.291046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8285.946876] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8299.642889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8299.874651] Lustre: 232561:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8299.965075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8299.971661] Lustre: Skipped 1 previous similar message [ 8300.102664] Lustre: lustre-MDT0001: new disk, initializing [ 8300.251935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.298781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8300.320637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8306.759609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8317.593418] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8323.384993] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8334.145845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8334.476850] Lustre: lustre-OST0000: new disk, initializing [ 8334.481781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8334.498550] Lustre: 234305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8334.703953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8342.575620] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8343.697545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8343.711324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8343.926944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8351.367885] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8354.616969] LustreError: 234685:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8364.726638] LustreError: 234685:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8364.751623] LustreError: 234684:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8366.984737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8367.249305] Lustre: lustre-OST0001: new disk, initializing [ 8367.258899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8367.268534] Lustre: 235308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8367.407426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8372.864123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8372.882224] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8373.057888] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8374.823890] LustreError: 234684:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8375.878865] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8386.595401] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8389.311835] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8393.700679] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8393.735820] Lustre: Skipped 2 previous similar messages [ 8393.745615] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8393.749039] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8393.759218] Lustre: Skipped 3 previous similar messages [ 8398.280588] Lustre: server umount lustre-OST0001 complete [ 8398.830202] LustreError: 234290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8398.874930] LustreError: 234290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8419.300838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8419.305214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8419.321527] Lustre: Skipped 2 previous similar messages [ 8419.329994] Lustre: Skipped 1 previous similar message [ 8423.665720] Lustre: server umount lustre-OST0000 complete [ 8429.537353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8429.554983] Lustre: Skipped 1 previous similar message [ 8429.555156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8429.573695] Lustre: Skipped 2 previous similar messages [ 8439.778420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8439.798782] Lustre: Skipped 3 previous similar messages [ 8443.359281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8443.562950] Lustre: server umount lustre-MDT0000 complete [ 8448.378233] LustreError: 232552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225755 with bad export cookie 2683324032929266415 [ 8448.383917] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.397986] LustreError: 232552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8455.250516] Lustre: server umount lustre-MDT0001 complete [ 8469.679871] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8473.253877] Key type lgssc unregistered [ 8473.696825] LNet: 237211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8473.706553] LNetError: 237211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8473.727026] LNet: Removed LNI 192.168.202.136@tcp [ 8474.905899] Key type .llcrypt unregistered [ 8474.908209] Key type ._llcrypt unregistered [ 8527.491720] Key type ._llcrypt registered [ 8527.494723] Key type .llcrypt registered [ 8527.596224] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8529.190435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8529.423492] alg: No test for adler32 (adler32-zlib) [ 8530.529425] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 8530.793681] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8532.519162] Key type lgssc registered [ 8534.038847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8544.413988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8544.439037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8545.958884] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8545.984680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8546.210537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8552.127990] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8565.204602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.459585] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8565.541724] Lustre: 239408:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8565.907248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8572.936507] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8580.290592] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8584.723332] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8593.774318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8593.912161] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8594.085372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.466696] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8603.893991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8609.053409] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8670.688571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8670.702304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8670.712834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8670.728414] Lustre: Skipped 1 previous similar message [ 8670.756717] Lustre: Skipped 1 previous similar message [ 8672.156062] Lustre: server umount lustre-OST0000 complete [ 8678.646508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8696.632752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8697.291622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.440353] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8710.649959] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8710.670105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8710.671260] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8710.687325] Lustre: Skipped 1 previous similar message [ 8715.663171] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8760.996319] Lustre: server umount lustre-OST0000 complete [ 8765.408698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8765.419139] LustreError: Skipped 1 previous similar message [ 8765.421229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.452157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8766.945451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8766.946359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8772.065406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8772.088747] Lustre: Skipped 2 previous similar messages [ 8775.361866] Lustre: server umount lustre-MDT0000 complete [ 8777.185393] Lustre: 238728:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226068/real 1781226068] req@00000000da0c5fdb x1867750663333888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781226084 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8777.214892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.300298] LustreError: 241054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226086 with bad export cookie 13605397718715439807 [ 8779.309532] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8779.312586] LustreError: 241054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8779.685986] Lustre: server umount lustre-MDT0001 complete [ 8793.003547] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8796.797488] Key type lgssc unregistered [ 8797.196900] LNet: 244384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8797.205975] LNetError: 244384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8797.234394] LNet: Removed LNI 192.168.202.136@tcp [ 8798.352540] Key type .llcrypt unregistered [ 8798.355270] Key type ._llcrypt unregistered [ 8826.437543] Key type ._llcrypt registered [ 8826.439652] Key type .llcrypt registered [ 8826.577599] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8827.977044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8828.106871] alg: No test for adler32 (adler32-zlib) [ 8829.210307] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 8829.431848] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8831.120822] Key type lgssc registered [ 8832.535263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8843.596636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8843.629335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8845.503655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8852.599535] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8856.032959] LustreError: 245785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8856.103833] LustreError: 245785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8861.157506] LustreError: 245786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.280908] LustreError: 245785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.469199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8867.013763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8873.227340] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8883.117372] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8888.683468] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8897.611564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8898.121661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.466389] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8915.601615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8915.714656] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8946.678199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8946.683084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.689607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.694243] Lustre: Skipped 1 previous similar message [ 8949.471864] Lustre: server umount lustre-OST0000 complete [ 8954.339178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8954.347703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.356085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8954.359178] Lustre: Skipped 1 previous similar message [ 8956.897638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.899190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.916752] Lustre: Skipped 1 previous similar message [ 8960.421300] Lustre: server umount lustre-MDT0000 complete [ 8962.025289] LustreError: 245786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8962.055573] LustreError: 245786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8964.892827] LustreError: 245766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226271 with bad export cookie 9172615820455340002 [ 8964.919475] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.366115] Lustre: server umount lustre-MDT0001 complete [ 8983.543200] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8987.817938] Key type lgssc unregistered [ 8988.192743] LNet: 249739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8988.211293] LNetError: 249739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8988.244182] LNet: Removed LNI 192.168.202.136@tcp [ 8989.296971] Key type .llcrypt unregistered [ 8989.299181] Key type ._llcrypt unregistered [ 9010.984234] Key type ._llcrypt registered [ 9010.986913] Key type .llcrypt registered [ 9011.101992] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9012.181987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9012.592993] alg: No test for adler32 (adler32-zlib) [ 9013.661821] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 9013.975388] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9015.687173] Key type lgssc registered [ 9017.069260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9025.524819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9025.558392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.198433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9032.176275] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9036.256492] LustreError: 250946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9036.300121] LustreError: 250946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9041.386975] LustreError: 250947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.016300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9042.419644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9047.952284] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9055.608158] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9060.627432] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9068.626883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9069.179042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9071.216863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9077.052800] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9085.306753] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9102.304704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9102.320444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.360044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9103.337521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.348171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9103.358415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9103.389854] Lustre: Skipped 1 previous similar message [ 9104.669892] Lustre: server umount lustre-OST0000 complete [ 9113.568869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.574451] Lustre: Skipped 1 previous similar message [ 9113.574865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9113.589307] Lustre: Skipped 1 previous similar message [ 9116.023419] Lustre: server umount lustre-MDT0000 complete [ 9118.691745] LustreError: 253098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9118.728391] LustreError: 253098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9120.872213] LustreError: 250925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226427 with bad export cookie 5796838025901898758 [ 9120.886395] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9121.241423] Lustre: server umount lustre-MDT0001 complete [ 9133.663375] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9136.377298] Key type lgssc unregistered [ 9136.648658] LNet: 254269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9136.659278] LNetError: 254269:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9136.686441] LNet: Removed LNI 192.168.202.136@tcp [ 9137.622173] Key type .llcrypt unregistered [ 9137.624064] Key type ._llcrypt unregistered [ 9166.497313] Key type ._llcrypt registered [ 9166.501509] Key type .llcrypt registered [ 9166.726524] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9168.114961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9168.195733] alg: No test for adler32 (adler32-zlib) [ 9169.405826] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 9169.636773] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9171.288349] Key type lgssc registered [ 9173.304777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9185.287412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9185.336509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.367410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.830296] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9197.552781] LustreError: 255672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.585678] LustreError: 255672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9202.659180] LustreError: 255673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9202.995870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9203.432483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9208.305613] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9215.077487] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9220.233821] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9228.965252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9229.308773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9232.400981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9237.292657] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9243.679986] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9278.432178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9278.442343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9278.470617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9279.456743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9279.458921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9281.948674] Lustre: server umount lustre-OST0000 complete [ 9289.703038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.724854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9289.740406] Lustre: Skipped 2 previous similar messages [ 9293.151545] Lustre: server umount lustre-MDT0000 complete [ 9294.817075] LustreError: 255672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9294.841702] LustreError: 255672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9297.429644] LustreError: 255653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226604 with bad export cookie 6331034864232168851 [ 9297.437216] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9297.441683] LustreError: 255653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9297.937793] Lustre: server umount lustre-MDT0001 complete [ 9311.664472] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9314.718212] Key type lgssc unregistered [ 9315.044182] LNet: 259624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9315.052649] LNetError: 259624:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9315.071073] LNet: Removed LNI 192.168.202.136@tcp [ 9316.333424] Key type .llcrypt unregistered [ 9316.335560] Key type ._llcrypt unregistered [ 9338.145757] Key type ._llcrypt registered [ 9338.152589] Key type .llcrypt registered [ 9338.312938] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9340.344913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9340.396727] alg: No test for adler32 (adler32-zlib) [ 9341.464408] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 9341.690531] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9343.439275] Key type lgssc registered [ 9345.199930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9353.352773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9353.387907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.149199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.939650] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9364.471962] LustreError: 260846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.514195] LustreError: 260846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9369.573441] LustreError: 260848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9371.307030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9371.688652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9376.569344] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9383.962385] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9390.259813] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9400.552562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9400.912921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.928919] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9416.355058] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9424.459274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9441.762017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9441.779671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9441.798665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9444.832544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9444.838551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9444.848619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.885412] Lustre: Skipped 1 previous similar message [ 9447.566156] Lustre: server umount lustre-OST0000 complete [ 9453.536724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9453.553013] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.596114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9458.876643] Lustre: server umount lustre-MDT0000 complete [ 9460.192941] LustreError: 260849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9460.217575] LustreError: 260849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9463.576201] LustreError: 260813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226770 with bad export cookie 7204776937107501448 [ 9463.584412] LustreError: 260813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9463.587806] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9463.961362] Lustre: server umount lustre-MDT0001 complete [ 9478.587537] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9480.821551] Key type lgssc unregistered [ 9481.135291] LNet: 264216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9481.146513] LNetError: 264216:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9481.159751] LNet: Removed LNI 192.168.202.136@tcp [ 9481.770442] Key type .llcrypt unregistered [ 9481.772700] Key type ._llcrypt unregistered [ 9495.820830] Key type ._llcrypt registered [ 9495.823866] Key type .llcrypt registered [ 9495.942773] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9516.133835] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9534.280726] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9535.546511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9535.655705] alg: No test for adler32 (adler32-zlib) [ 9536.698112] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 9537.100177] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9538.872593] Key type lgssc registered [ 9539.826739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9586.809867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9586.908199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.690258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9588.802685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9589.028967] Lustre: lustre-MDT0000: new disk, initializing [ 9589.226964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9589.252788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9594.569458] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9607.454351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.538857] Lustre: 267721:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9607.613675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9607.619287] Lustre: Skipped 1 previous similar message [ 9607.707162] Lustre: lustre-MDT0001: new disk, initializing [ 9607.862765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9607.903920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9607.921972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9613.687386] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9622.329289] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9627.064252] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9635.864889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9636.181833] Lustre: lustre-OST0000: new disk, initializing [ 9636.184996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9636.203042] Lustre: 269462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9636.332282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9642.033642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9642.050792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9642.168047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9643.804142] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9654.622885] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9661.635961] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9662.178253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9667.890449] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9668.210299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9672.674767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9672.719227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9672.721593] Lustre: Skipped 1 previous similar message [ 9672.746808] Lustre: Skipped 1 previous similar message [ 9677.177269] Lustre: server umount lustre-OST0000 complete [ 9682.913299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.931259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9687.901835] Lustre: server umount lustre-MDT0000 complete [ 9692.394593] LustreError: 268499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226999 with bad export cookie 11745802692127833732 [ 9692.397520] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9692.407448] LustreError: 268499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9692.808182] Lustre: server umount lustre-MDT0001 complete [ 9703.176267] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:16:47 (1781227007) [ 9712.113380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.466666] LustreError: 271729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9712.485170] LustreError: 271729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9712.558422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.137348] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9717.731241] LustreError: 271730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.861415] LustreError: 271729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9727.243626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.854756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9733.232895] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9740.733901] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9745.563628] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9753.937913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9754.414273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9763.399783] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9770.195198] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9788.400271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.428111] Lustre: Skipped 1 previous similar message [ 9788.440385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9788.448098] LustreError: 273325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.710519] Lustre: server umount lustre-OST0000 complete [ 9798.624822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.629202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9798.663989] Lustre: Skipped 2 previous similar messages [ 9798.698551] Lustre: Skipped 2 previous similar messages [ 9801.231958] Lustre: server umount lustre-MDT0000 complete [ 9803.746325] LustreError: 271726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.767738] LustreError: 271726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9805.968689] LustreError: 271711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227112 with bad export cookie 11745802692127834614 [ 9805.998163] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9806.582843] Lustre: server umount lustre-MDT0001 complete [ 9820.669192] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9824.449928] Key type lgssc unregistered [ 9824.877978] LNet: 274931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9824.892734] LNetError: 274931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9824.913329] LNet: Removed LNI 192.168.202.136@tcp [ 9825.984239] Key type .llcrypt unregistered [ 9825.986486] Key type ._llcrypt unregistered [ 9859.849470] Key type ._llcrypt registered [ 9859.862106] Key type .llcrypt registered [ 9860.048294] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9878.071944] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9879.280945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9879.500517] alg: No test for adler32 (adler32-zlib) [ 9880.715694] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [ 9880.965759] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9882.707617] Key type lgssc registered [ 9884.639439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9941.645748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9941.726774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.083481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9943.131705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9943.269605] Lustre: lustre-MDT0000: new disk, initializing [ 9943.372334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9943.387963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9949.082864] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9962.609498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9962.755358] Lustre: 278727:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9962.856833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9962.863544] Lustre: Skipped 1 previous similar message [ 9962.970555] Lustre: lustre-MDT0001: new disk, initializing [ 9963.073107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9963.143853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9963.180963] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9968.227648] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9977.390642] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9982.174811] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9990.022786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9990.414228] Lustre: lustre-OST2710: new disk, initializing [ 9990.417901] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9990.424141] Lustre: 280472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9990.543671] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9995.889524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9995.906637] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9996.058397] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9996.394805] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10004.873030] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [10012.545306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10012.680807] Lustre: lustre-OST03e8: new disk, initializing [10012.689552] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [10012.700388] Lustre: 281420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10012.807395] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10019.428590] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10019.960041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10019.975471] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10020.181560] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [10025.819973] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [10033.500131] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10033.699581] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10038.800270] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10039.049930] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10045.107381] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10045.295389] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10050.773418] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10051.021397] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10060.256950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10060.261486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.274965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10061.282538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.286538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10061.308721] Lustre: Skipped 1 previous similar message [10061.324297] Lustre: Skipped 3 previous similar messages [10063.443208] Lustre: server umount lustre-MDT0000 complete [10067.280326] LustreError: 279504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227374 with bad export cookie 9207313120091129574 [10067.286607] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10067.295931] LustreError: 279504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10067.744725] Lustre: server umount lustre-MDT0001 complete [10082.438174] Lustre: server umount lustre-OST2710 complete [10087.329258] Lustre: 276522:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781227378/real 1781227378] req@00000000427f06c6 x1867752078908416/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781227394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10087.411076] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10087.489488] Lustre: Skipped 1 previous similar message [10088.523417] Lustre: server umount lustre-OST03e8 complete [10116.697850] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10132.899953] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10176.111897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.366568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10176.400292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10176.490924] Lustre: lustre-MDT0000: new disk, initializing [10176.576062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10176.603959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10181.057770] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10193.221780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.351770] Lustre: 286477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10193.405154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10193.410399] Lustre: Skipped 1 previous similar message [10193.495877] Lustre: lustre-MDT0001: new disk, initializing [10193.636510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10193.686902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10193.701450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10198.414931] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10206.743365] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10211.750945] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10220.234614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10220.595558] Lustre: lustre-OST0000: new disk, initializing [10220.599366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10220.606305] Lustre: 288221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10220.717946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10227.204323] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10227.332501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10227.353233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10227.528076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10235.948339] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10240.675361] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10240.918641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10246.298856] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10246.587911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10253.284498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10253.286253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10253.294022] LustreError: Skipped 1 previous similar message [10253.305060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10253.317183] Lustre: Skipped 1 previous similar message [10253.356198] Lustre: Skipped 1 previous similar message [10255.326366] Lustre: server umount lustre-OST0000 complete [10260.448761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10260.458903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.483356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10268.651449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10268.659159] Lustre: Skipped 3 previous similar messages [10274.271121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10274.528651] Lustre: server umount lustre-MDT0000 complete [10278.483177] LustreError: 286471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227585 with bad export cookie 9207313120091131268 [10278.499161] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10278.501021] LustreError: 286471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10278.965905] Lustre: server umount lustre-MDT0001 complete [10288.947475] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:26:33 (1781227593) [10290.892891] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10293.033769] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:26:37 (1781227597) [10297.238643] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10326.574148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10326.986949] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10327.028908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10327.234395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.690554] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10341.231572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.367121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10341.399826] Lustre: 291918:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10345.927529] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10368.619686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10368.835794] Lustre: Denying initial registration attempt from nid 192.168.202.136@tcp, specified as failover [10368.850217] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10368.861569] LustreError: 293337:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [10368.873170] LustreError: 293337:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [10368.877879] LustreError: 293337:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10368.966501] Lustre: server umount lustre-OST0000 complete [10368.969446] LustreError: 293337:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [10377.697461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10377.706508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10377.714110] Lustre: Skipped 1 previous similar message [10377.717889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10377.721315] Lustre: Skipped 2 previous similar messages [10382.667515] Lustre: server umount lustre-MDT0000 complete [10385.378220] LustreError: 291923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.398672] LustreError: 291923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10386.879804] LustreError: 291908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227693 with bad export cookie 9207313120091132150 [10386.883371] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10386.893158] LustreError: 291908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10387.144689] Lustre: server umount lustre-MDT0001 complete [10400.247429] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10402.801567] Key type lgssc unregistered [10403.116832] LNet: 294442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10403.135970] LNetError: 294442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10403.156921] LNet: Removed LNI 192.168.202.136@tcp [10404.553276] Key type .llcrypt unregistered [10404.559105] Key type ._llcrypt unregistered [10417.259853] Key type ._llcrypt registered [10417.262226] Key type .llcrypt registered [10417.394246] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10418.488056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10418.611337] alg: No test for adler32 (adler32-zlib) [10419.625772] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [10419.816547] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10421.487273] Key type lgssc registered [10422.924590] Lustre: Echo OBD driver; http://www.lustre.org/ [10480.186725] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10491.585107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10491.595936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10492.905804] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10492.957865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10493.143081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10498.018618] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10508.327489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10508.571825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10508.632413] Lustre: 297178:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10509.044242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10513.982797] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10520.608962] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10525.372458] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10546.357520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10546.836215] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10547.291866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10554.843186] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10560.987877] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10565.602033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10565.608700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.637736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10566.116022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10566.134073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10566.147558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10566.157362] Lustre: Skipped 1 previous similar message [10571.245747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.254113] Lustre: Skipped 1 previous similar message [10576.357974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10577.381356] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10577.729378] Lustre: server umount lustre-OST0000 complete [10581.472285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10581.472893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10581.484581] Lustre: Skipped 1 previous similar message [10581.495266] Lustre: Skipped 2 previous similar messages [10588.012484] Lustre: server umount lustre-MDT0000 complete [10592.580496] LustreError: 297170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227899 with bad export cookie 2268027455786552027 [10592.605333] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10593.159611] Lustre: server umount lustre-MDT0001 complete [10605.509532] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10609.709857] Key type lgssc unregistered [10610.059152] LNet: 300394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10610.070541] LNetError: 300394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10610.084160] LNet: Removed LNI 192.168.202.136@tcp [10611.068128] Key type .llcrypt unregistered [10611.070389] Key type ._llcrypt unregistered [10638.151374] Key type ._llcrypt registered [10638.152796] Key type .llcrypt registered [10638.245542] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10639.269765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10639.299833] alg: No test for adler32 (adler32-zlib) [10640.300754] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [10640.498179] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10642.255543] Key type lgssc registered [10643.508790] Lustre: Echo OBD driver; http://www.lustre.org/ [10651.220562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10651.250936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.619628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10656.258459] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10662.890093] LustreError: 301795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10662.914143] LustreError: 301795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10664.923955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.419446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10669.986491] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10675.973103] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10679.971278] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10690.936476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10691.642511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10698.622409] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10705.799467] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10723.821954] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [10726.374018] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10726.384134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10726.398814] Lustre: Skipped 1 previous similar message [10728.698419] Lustre: server umount lustre-MDT0000 complete [10728.921714] LustreError: 301792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.507832] LustreError: 303625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10731.550114] LustreError: 303625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10734.027066] LustreError: 301790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.471741] LustreError: 301777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228041 with bad export cookie 16006222883778696529 [10734.484629] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10736.609027] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10736.624100] Lustre: Skipped 1 previous similar message [10736.639536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10736.653601] Lustre: Skipped 2 previous similar messages [10739.145586] LustreError: 303625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10739.164092] LustreError: 303625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10739.187285] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [10740.890389] Lustre: server umount lustre-MDT0001 complete [10748.340626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10760.117991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.615751] LustreError: 301105:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000856dc948 x1867752875800576/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10780.018666] LustreError: 303402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.038957] LustreError: 303402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10780.110476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10780.164122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10785.363480] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10794.372144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10795.099136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10796.141926] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10801.292091] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10809.100881] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10811.592566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10816.871882] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10818.903761] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10825.701418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10825.719242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.739661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10825.746203] Lustre: Skipped 1 previous similar message [10826.218332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10831.271817] Lustre: server umount lustre-OST0000 complete [10835.937910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10835.946393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10835.966017] Lustre: Skipped 1 previous similar message [10835.981500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10835.990284] Lustre: Skipped 3 previous similar messages [10841.836427] Lustre: server umount lustre-MDT0000 complete [10842.598141] LustreError: 305074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.632584] LustreError: 305074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10845.634874] LustreError: 301776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228152 with bad export cookie 16006222883778704397 [10845.640269] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10845.649081] LustreError: 301776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10845.902325] Lustre: server umount lustre-MDT0001 complete [10858.540893] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10861.428822] Key type lgssc unregistered [10861.830599] LNet: 307506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10861.840604] LNetError: 307506:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10862.886708] LNet: Removed LNI 192.168.202.136@tcp [10863.793908] Key type .llcrypt unregistered [10863.796288] Key type ._llcrypt unregistered [10894.526538] Key type ._llcrypt registered [10894.528904] Key type .llcrypt registered [10894.670156] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10896.613328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10896.648975] alg: No test for adler32 (adler32-zlib) [10897.818912] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [10898.112401] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10899.855721] Key type lgssc registered [10901.341800] Lustre: Echo OBD driver; http://www.lustre.org/ [10910.206455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10910.224718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10911.846594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10916.081173] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10921.952087] LustreError: 308907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.978890] LustreError: 308907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10925.368939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.832215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10930.633792] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10936.710134] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10941.812303] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10946.530496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10946.532185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.573479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10950.689570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10950.698621] Lustre: Skipped 2 previous similar messages [10955.744901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10955.754033] Lustre: Skipped 1 previous similar message [10960.351307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10960.695085] Lustre: server umount lustre-MDT0000 complete [10960.864764] LustreError: 308906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.892843] LustreError: 308906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10965.132604] LustreError: 308886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228272 with bad export cookie 10306843929263951602 [10965.136586] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10965.144124] LustreError: 308886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10965.551712] Lustre: server umount lustre-MDT0001 complete [10983.350489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10983.642619] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10983.668390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10983.820650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.594988] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10999.373482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.555430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10999.632858] Lustre: 311269:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11000.003493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11005.373089] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11012.535405] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11018.772501] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11030.889180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11030.979352] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11030.984264] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11030.988830] LustreError: 312929:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [11030.992568] LustreError: 312929:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [11030.996399] LustreError: 312929:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11031.040691] Lustre: server umount lustre-OST0000 complete [11031.042682] LustreError: 312929:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [11041.773304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11041.998954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11042.161788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11049.298241] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11051.792870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11056.246611] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11065.386186] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11065.493189] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11065.568603] Lustre: lustre-OST0001: new disk, initializing [11065.573743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11065.582641] Lustre: 314358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11065.700757] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11071.813669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11073.113148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11073.128801] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11073.257234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11078.283234] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11083.753643] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.755763] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11083.773209] Lustre: Skipped 1 previous similar message [11083.779383] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11083.789847] Lustre: Skipped 1 previous similar message [11088.167948] Lustre: server umount lustre-OST0001 complete [11088.876541] LustreError: 313396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.898546] LustreError: 313396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11092.966221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11092.971314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11092.976394] LustreError: Skipped 1 previous similar message [11093.000537] Lustre: Skipped 1 previous similar message [11093.011548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11093.024855] Lustre: Skipped 1 previous similar message [11093.988724] LustreError: 313849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11094.016918] LustreError: 313849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11098.759693] Lustre: server umount lustre-OST0000 complete [11104.227146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.239277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11104.256461] Lustre: Skipped 2 previous similar messages [11104.270576] Lustre: Skipped 3 previous similar messages [11109.638124] Lustre: server umount lustre-MDT0000 complete [11113.863472] LustreError: 311262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228420 with bad export cookie 10306843929263952155 [11113.872536] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.879167] LustreError: 311262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11114.169490] Lustre: server umount lustre-MDT0001 complete [11126.680813] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11129.969392] Key type lgssc unregistered [11130.380731] LNet: 316185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11130.386254] LNetError: 316185:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11130.399791] LNet: Removed LNI 192.168.202.136@tcp [11131.305700] Key type .llcrypt unregistered [11131.315936] Key type ._llcrypt unregistered [11213.940328] Key type ._llcrypt registered [11213.943475] Key type .llcrypt registered [11214.044179] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [11233.461730] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11234.514522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11234.697024] alg: No test for adler32 (adler32-zlib) [11235.843804] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [11236.162086] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11237.863190] Key type lgssc registered [11239.631041] Lustre: Echo OBD driver; http://www.lustre.org/ [11279.597831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11279.644688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11281.070725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11281.098744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11281.156475] Lustre: lustre-MDT0000: new disk, initializing [11281.213896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11281.229335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11286.481346] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11298.336875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.606454] Lustre: 321089:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11298.693429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11298.697138] Lustre: Skipped 1 previous similar message [11298.760612] Lustre: lustre-MDT0001: new disk, initializing [11298.852667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11298.886858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11298.900220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11304.066022] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11312.241872] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11316.675278] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11323.796763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.032766] Lustre: lustre-OST0000: new disk, initializing [11324.036108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11324.041580] Lustre: 322833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11324.121646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11330.184669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11330.196036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11330.411521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11331.045225] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11339.059892] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11343.684954] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11343.910735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11349.000680] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11349.221953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11356.139386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11356.142171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.148893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11356.188294] Lustre: Skipped 1 previous similar message [11357.716135] Lustre: server umount lustre-OST0000 complete [11365.347941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11365.355858] LustreError: Skipped 1 previous similar message [11365.371583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11365.416898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11365.435211] Lustre: Skipped 1 previous similar message [11366.379460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11368.571660] Lustre: server umount lustre-MDT0000 complete [11372.473190] LustreError: 321081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228679 with bad export cookie 14557503300747329768 [11372.481375] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.485942] LustreError: 321081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11372.790218] Lustre: server umount lustre-MDT0001 complete [11383.815710] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:44:48 (1781228688) [11394.724085] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:44:58 (1781228698) [11404.945687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11405.595575] LustreError: 325440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.620935] LustreError: 325440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11405.721108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11410.045749] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11410.915292] LustreError: 325441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.039765] LustreError: 325440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.154688] LustreError: 325441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11422.776449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11423.208346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11429.193392] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11435.280862] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11440.359214] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11448.579591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11449.092158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11455.278600] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11461.344303] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11475.174939] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11477.709213] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11479.679965] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11481.935800] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11484.452647] Lustre: Failing over lustre-MDT0000 [11484.642119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11484.653255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11484.675526] LustreError: 325441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.696833] LustreError: 325441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11484.780937] Lustre: server umount lustre-MDT0000 complete [11493.344171] LustreError: 325440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11493.383418] LustreError: 325440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11504.440467] Lustre: 319315:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781228795/real 1781228795] req@000000007fff6f24 x1867753500780544/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781228811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11504.498433] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11506.970505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11509.732292] LustreError: 326152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.785527] LustreError: 326152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [11514.891469] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xca06a41f819f0874 [11514.922279] Lustre: MGC192.168.202.136@tcp: Connection restored to 0@lo (at 0@lo) [11515.250828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11515.337243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11516.368933] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11520.506308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11520.584942] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11520.662519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11522.022593] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11531.519777] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11534.410749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11539.022434] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11541.517202] Lustre: Failing over lustre-MDT0000 [11542.006427] LustreError: 325437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.022884] LustreError: 325437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11542.215911] Lustre: server umount lustre-MDT0000 complete [11546.092947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11546.096641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.126489] Lustre: Skipped 4 previous similar messages [11561.939628] Lustre: 319313:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781228852/real 1781228852] req@00000000dd64e179 x1867753500803712/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781228868 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11561.981589] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11562.148697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11571.168756] LustreError: 319311:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dd64e179 x1867753500808448/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11571.656560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11571.805696] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11572.690756] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11576.850337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11576.858920] Lustre: Skipped 2 previous similar messages [11576.910453] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11576.986157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11578.921164] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11588.138490] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11590.006864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11597.282729] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.288373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11597.306625] Lustre: Skipped 1 previous similar message [11597.316591] Lustre: Skipped 2 previous similar messages [11602.402776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.417321] Lustre: Skipped 4 previous similar messages [11603.508931] Lustre: server umount lustre-MDT0000 complete [11607.527701] LustreError: 326152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.574552] LustreError: 326152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [11609.061127] LustreError: 326159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228915 with bad export cookie 14557503300747332491 [11609.065124] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11609.083284] LustreError: 326159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11609.902163] Lustre: server umount lustre-MDT0001 complete [11624.619565] Lustre: server umount lustre-OST0000 complete [11644.281603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.985960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11650.120596] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11662.019152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11666.701452] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11673.497781] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11678.400718] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11687.357564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11689.989287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11696.283363] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11703.362742] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11710.434431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11710.438680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.439794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11710.476925] Lustre: Skipped 2 previous similar messages [11720.677095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11720.696628] Lustre: Skipped 3 previous similar messages [11724.255165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11724.453161] Lustre: server umount lustre-OST0000 complete [11728.876511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11741.152946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11741.162518] Lustre: Skipped 7 previous similar messages [11743.201226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11743.773079] Lustre: server umount lustre-MDT0000 complete [11746.280680] LustreError: 331245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11746.313254] LustreError: 331245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11749.686346] LustreError: 331226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229056 with bad export cookie 14557503300747333233 [11749.699901] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11750.229905] Lustre: server umount lustre-MDT0001 complete [11765.673523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11769.792145] Key type lgssc unregistered [11770.329878] LNet: 334424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11770.374356] LNetError: 334424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11771.428975] LNet: Removed LNI 192.168.202.136@tcp [11772.635092] Key type .llcrypt unregistered [11772.643521] Key type ._llcrypt unregistered [11807.308715] Key type ._llcrypt registered [11807.311867] Key type .llcrypt registered [11807.439932] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11809.802604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11809.828248] alg: No test for adler32 (adler32-zlib) [11811.141712] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [11811.573369] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11813.335171] Key type lgssc registered [11815.423395] Lustre: Echo OBD driver; http://www.lustre.org/ [11825.936092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11825.951194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11827.620932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11832.728603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11837.926407] LustreError: 335930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.967778] LustreError: 335930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11842.988217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11843.056104] LustreError: 335929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.591803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11850.359682] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11857.724990] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11862.641888] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11870.497285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11870.980230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11873.016830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11878.321317] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11886.071561] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11898.857547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11898.859559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11898.871918] Lustre: Skipped 2 previous similar messages [11898.875957] Lustre: Skipped 1 previous similar message [11903.114588] Lustre: server umount lustre-MDT0000 complete [11903.972610] LustreError: 335924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11904.019722] LustreError: 335924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11908.421743] LustreError: 335909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229215 with bad export cookie 9844004810570724556 [11908.430409] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11908.448764] LustreError: 335909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11908.976490] Lustre: server umount lustre-MDT0001 complete [11922.643631] Lustre: server umount lustre-OST0000 complete [11938.442593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11939.277245] LustreError: 339304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11939.342735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11944.422664] LustreError: 339305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.805209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11954.656836] LustreError: 339304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.675518] LustreError: 339304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11955.051954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11955.505154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11961.709701] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11969.436310] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11974.025856] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11982.360662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11982.957864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11985.020749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11990.108663] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11996.477679] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12010.977379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12010.987611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.992308] Lustre: Skipped 1 previous similar message [12011.012572] Lustre: Skipped 3 previous similar messages [12016.099317] LustreError: 339304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12016.111235] LustreError: 339304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12017.864726] Lustre: server umount lustre-MDT0000 complete [12022.045475] LustreError: 339285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229328 with bad export cookie 9844004810570725669 [12022.053203] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12022.078695] LustreError: 339285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12022.486535] Lustre: server umount lustre-MDT0001 complete [12036.697359] Lustre: server umount lustre-OST0000 complete [12052.085932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12052.876658] LustreError: 342876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12052.926935] LustreError: 342876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12053.057208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12058.432161] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12069.482304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.301769] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12083.590552] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12087.826804] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12095.167487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12095.665384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12095.677364] Lustre: Skipped 1 previous similar message [12097.754647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12102.684076] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12109.556126] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12118.496340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12118.508304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12118.521239] Lustre: Skipped 1 previous similar message [12118.526553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12120.545517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12120.547204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12120.552744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12120.578221] Lustre: Skipped 1 previous similar message [12121.642800] Lustre: server umount lustre-OST0000 complete [12126.692825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12126.709806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12126.752413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12132.889491] Lustre: server umount lustre-MDT0000 complete [12135.907781] LustreError: 342877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.919369] LustreError: 342877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12137.140311] LustreError: 342858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229444 with bad export cookie 9844004810570726691 [12137.143485] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12137.146502] LustreError: 342858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12137.466247] Lustre: server umount lustre-MDT0001 complete [12149.764643] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12152.919569] Key type lgssc unregistered [12153.257820] LNet: 346055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12153.273187] LNetError: 346055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12153.290796] LNet: Removed LNI 192.168.202.136@tcp [12154.087373] Key type .llcrypt unregistered [12154.089534] Key type ._llcrypt unregistered [12180.352368] Key type ._llcrypt registered [12180.354609] Key type .llcrypt registered [12180.483904] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12182.047484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12182.118046] alg: No test for adler32 (adler32-zlib) [12183.185970] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [12183.400514] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12185.079234] Key type lgssc registered [12186.300953] Lustre: Echo OBD driver; http://www.lustre.org/ [12194.447612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12194.488125] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12194.501146] LustreError: 347483:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12194.525573] LustreError: 347483:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12194.537794] LustreError: 347483:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12194.546464] LustreError: 347483:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12194.560133] LustreError: 347483:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [12202.760524] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12202.771168] LustreError: 347880:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12202.798937] LustreError: 347880:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12202.804881] LustreError: 347880:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12202.810984] LustreError: 347880:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12202.817282] LustreError: 347880:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [12224.586364] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12227.637953] Key type lgssc unregistered [12227.921886] LNet: 348866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12227.924742] LNetError: 348866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12227.940651] LNet: Removed LNI 192.168.202.136@tcp [12228.804710] Key type .llcrypt unregistered [12228.805924] Key type ._llcrypt unregistered [12248.934609] Key type ._llcrypt registered [12248.938567] Key type .llcrypt registered [12249.043906] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [12266.118371] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12266.992362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12267.015141] alg: No test for adler32 (adler32-zlib) [12268.099666] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [12268.328962] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12270.119685] Key type lgssc registered [12271.383092] Lustre: Echo OBD driver; http://www.lustre.org/ [12307.640350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12307.724550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.244378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12309.343213] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12309.492366] Lustre: lustre-MDT0000: new disk, initializing [12309.609947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12309.648383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12314.660157] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12326.432785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.640782] Lustre: 351990:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12326.709829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12326.719259] Lustre: Skipped 1 previous similar message [12326.812265] Lustre: lustre-MDT0001: new disk, initializing [12326.960732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12327.026318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12327.042703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12333.213824] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12342.208281] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12346.689474] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12355.384235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12355.890546] Lustre: lustre-OST0000: new disk, initializing [12355.894545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12355.908156] Lustre: 353744:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12356.015488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12362.274632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12362.278249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12362.394086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12362.679964] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12371.698604] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12377.171296] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12377.421846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12381.941352] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12382.193535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12387.814081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12387.829987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.846495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12387.864120] Lustre: Skipped 1 previous similar message [12391.603766] Lustre: server umount lustre-OST0000 complete [12398.067612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.086888] Lustre: Skipped 1 previous similar message [12398.099135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.106632] Lustre: Skipped 1 previous similar message [12403.368048] Lustre: server umount lustre-MDT0000 complete [12407.613260] LustreError: 351983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229714 with bad export cookie 4638796254804024171 [12407.641475] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12407.647553] LustreError: 351983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12408.042725] Lustre: server umount lustre-MDT0001 complete [12419.779056] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12421.878095] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:02:06 (1781229726) [12429.561897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12430.249059] LustreError: 356058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.290207] LustreError: 356058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12430.385024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12434.993350] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12435.425728] LustreError: 356059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12440.546716] LustreError: 356058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.781538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12445.300081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12450.546986] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12457.384184] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12461.832805] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12468.428583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12468.780313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12474.906989] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12481.283526] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12489.176242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12489.442206] Lustre: lustre-OST0001: new disk, initializing [12489.448541] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12489.459436] Lustre: 358558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12489.584199] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12495.414088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12495.432057] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12495.580624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12496.228536] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12504.827621] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12511.217720] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12511.218142] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12511.231672] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12511.240631] Lustre: Skipped 2 previous similar messages [12511.255513] Lustre: Skipped 1 previous similar message [12515.472440] Lustre: server umount lustre-OST0001 complete [12516.337872] LustreError: 358845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.367833] LustreError: 358845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12521.450542] LustreError: 357655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12521.471545] LustreError: 357655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12531.709147] LustreError: 357656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.748721] LustreError: 357656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12536.804229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.813363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12536.828165] Lustre: Skipped 1 previous similar message [12536.850792] Lustre: Skipped 2 previous similar messages [12541.922818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12541.930714] Lustre: Skipped 1 previous similar message [12548.063490] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12548.493627] Lustre: server umount lustre-OST0000 complete [12557.280563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.306272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12557.310311] Lustre: Skipped 2 previous similar messages [12559.805968] Lustre: server umount lustre-MDT0000 complete [12562.402060] LustreError: 356058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12562.412907] LustreError: 356058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [12564.107923] LustreError: 356040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229870 with bad export cookie 4638796254804025046 [12564.135169] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12570.685970] Lustre: server umount lustre-MDT0001 complete [12585.072698] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12588.610587] Key type lgssc unregistered [12589.021781] LNet: 360421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12589.027959] LNetError: 360421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12589.059478] LNet: Removed LNI 192.168.202.136@tcp [12589.965517] Key type .llcrypt unregistered [12589.967593] Key type ._llcrypt unregistered [12657.251535] Key type ._llcrypt registered [12657.257132] Key type .llcrypt registered [12657.517684] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12659.885101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12660.046791] alg: No test for adler32 (adler32-zlib) [12661.155682] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [12661.511236] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12663.199259] Key type lgssc registered [12664.560805] Lustre: Echo OBD driver; http://www.lustre.org/ [12674.525169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12674.553385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12675.921811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12675.980235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12676.144923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12681.717748] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12690.679879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12690.781964] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12690.818492] Lustre: 363206:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12691.100272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12695.196783] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12701.783654] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12706.519254] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12708.837881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12708.865643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12708.891406] Lustre: Skipped 1 previous similar message [12713.957657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12713.971423] Lustre: Skipped 1 previous similar message [12719.073756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12719.086931] Lustre: Skipped 1 previous similar message [12723.167310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12723.328939] Lustre: server umount lustre-MDT0000 complete [12726.966361] LustreError: 363197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230033 with bad export cookie 3002978616668739505 [12726.982580] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12726.985830] LustreError: 363197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12727.379883] Lustre: server umount lustre-MDT0001 complete [12736.994788] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:07:22 (1781230042) [12744.702899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12745.037373] LustreError: 365492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12745.056885] LustreError: 365492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12745.121337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12749.370101] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12750.307428] LustreError: 365493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12755.432529] LustreError: 365492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12758.129606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12758.603780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12763.393432] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12769.768827] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12773.976971] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12782.526894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12782.882438] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12783.149823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12791.505556] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12800.546463] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12809.880313] Lustre: Permanently deactivating lustre-OST0000 [12809.925101] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12818.990575] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12820.351103] LustreError: 368142:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12820.368591] LustreError: 368142:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [12820.374638] LustreError: 368142:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12824.495741] LustreError: 368190:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [12824.509773] LustreError: 368190:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12833.512296] Lustre: server umount lustre-OST0000 complete [12836.928897] LustreError: 368442:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [12836.937490] LustreError: 368442:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12840.428937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12840.447963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12840.475732] Lustre: Skipped 1 previous similar message [12840.487379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12843.500937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12848.611237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12848.621787] Lustre: Skipped 3 previous similar messages [12849.720900] Lustre: server umount lustre-MDT0000 complete [12853.729257] LustreError: 365493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12853.742516] LustreError: 365493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12853.793838] LustreError: 365474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230160 with bad export cookie 3002978616668740086 [12853.799617] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12853.804116] LustreError: 365474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12864.133412] Lustre: server umount lustre-MDT0001 complete [12873.127503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12877.737441] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12880.427576] LustreError: 369586:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12887.000922] Lustre: 369925:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.136@tcp [12889.432917] Lustre: server umount MGS complete [12903.279294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12903.815910] LustreError: 370649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12903.849661] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12903.862523] Lustre: Skipped 1 previous similar message [12903.939912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12910.391853] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12914.157117] LustreError: 370650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.169552] LustreError: 370650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12921.662559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12922.141386] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12922.336424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12928.563488] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12936.909276] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12941.531538] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12944.211261] Lustre: Permanently reactivating lustre-OST0000 [12944.250081] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12944.260187] Lustre: Skipped 2 previous similar messages [12969.969284] LustreError: 372138:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12977.631457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12977.885235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12984.354177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12991.975381] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13000.679711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13000.691588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13000.695975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13000.703077] Lustre: Skipped 1 previous similar message [13000.722691] Lustre: Skipped 1 previous similar message [13001.461572] Lustre: server umount lustre-OST0000 complete [13009.386611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13009.397042] LustreError: Skipped 1 previous similar message [13009.406395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13012.722934] Lustre: server umount lustre-MDT0000 complete [13016.042024] LustreError: 371361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.060900] LustreError: 371361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13016.319399] LustreError: 370630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230323 with bad export cookie 3002978616668741136 [13016.327206] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13016.333869] LustreError: 370630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13016.855969] Lustre: server umount lustre-MDT0001 complete [13028.339323] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13031.286407] Key type lgssc unregistered [13031.638896] LNet: 374560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13031.643531] LNetError: 374560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13031.656424] LNet: Removed LNI 192.168.202.136@tcp [13032.499355] Key type .llcrypt unregistered [13032.501532] Key type ._llcrypt unregistered [13052.268690] Key type ._llcrypt registered [13052.271624] Key type .llcrypt registered [13052.364287] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13068.992900] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13070.217312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13070.336727] alg: No test for adler32 (adler32-zlib) [13071.503230] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [13071.768703] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13073.415208] Key type lgssc registered [13074.804912] Lustre: Echo OBD driver; http://www.lustre.org/ [13112.617990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13112.634345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13114.017699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13114.051821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13114.191556] Lustre: lustre-MDT0000: new disk, initializing [13114.308323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13114.350538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13119.077934] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13130.435201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13130.558611] Lustre: 377684:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13130.626752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13130.633035] Lustre: Skipped 1 previous similar message [13130.701514] Lustre: lustre-MDT0001: new disk, initializing [13130.766774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13130.801042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13130.816591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13135.489761] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13143.549196] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13147.317556] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13154.154542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13154.511795] Lustre: lustre-OST0000: new disk, initializing [13154.515929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13154.525278] Lustre: 379430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13154.678355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13160.610320] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13162.568668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13162.582576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13162.684916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13168.397773] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13172.919138] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13173.081785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13177.263235] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13177.434328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13183.458821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.464058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13183.465610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13183.481360] Lustre: Skipped 1 previous similar message [13183.513349] Lustre: Skipped 1 previous similar message [13185.579324] Lustre: server umount lustre-OST0000 complete [13192.160756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13192.171873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13192.208492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13193.696297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13193.701711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13193.735983] Lustre: Skipped 1 previous similar message [13195.847540] Lustre: server umount lustre-MDT0000 complete [13200.192680] LustreError: 377676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230507 with bad export cookie 15663539878850657683 [13200.205188] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13200.215787] LustreError: 377676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13200.724338] Lustre: server umount lustre-MDT0001 complete [13210.599480] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13212.376603] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:15:17 (1781230517) [13220.097478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13220.641647] LustreError: 381744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.663908] LustreError: 381744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13220.729580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13225.952919] LustreError: 381745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13226.425126] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13231.074111] LustreError: 381744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.203385] LustreError: 381745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.340649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13238.871920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13243.651885] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13249.410343] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13253.550716] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13261.476370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13261.884674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13267.883846] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13273.764645] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13275.429610] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13278.964210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13286.375528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13286.375750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13286.379244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13286.403289] Lustre: Skipped 1 previous similar message [13291.484385] Lustre: server umount lustre-OST0000 complete [13296.615463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13296.620306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13296.630223] Lustre: Skipped 1 previous similar message [13296.652921] Lustre: Skipped 2 previous similar messages [13302.319092] Lustre: server umount lustre-MDT0000 complete [13306.243159] LustreError: 381725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230613 with bad export cookie 15663539878850658558 [13306.251802] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13306.256253] LustreError: 381725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13306.718957] Lustre: server umount lustre-MDT0001 complete [13318.589385] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13321.347138] Key type lgssc unregistered [13321.614789] LNet: 385112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13321.624189] LNetError: 385112:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13321.637684] LNet: Removed LNI 192.168.202.136@tcp [13322.509924] Key type .llcrypt unregistered [13322.512867] Key type ._llcrypt unregistered [13345.113934] Key type ._llcrypt registered [13345.116387] Key type .llcrypt registered [13345.202985] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13345.428439] Key type .llcrypt unregistered [13345.431662] Key type ._llcrypt unregistered [13364.660650] Key type ._llcrypt registered [13364.665049] Key type .llcrypt registered [13364.791395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13366.343043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13366.384558] alg: No test for adler32 (adler32-zlib) [13367.541776] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [13367.719729] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13369.416099] Key type lgssc registered [13370.574928] Lustre: Echo OBD driver; http://www.lustre.org/ [13378.950159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13378.980698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13380.495835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13384.766310] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13390.816494] LustreError: 387290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.838402] LustreError: 387290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13393.662374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13394.020429] LustreError: 387291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13394.040452] LustreError: 387291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13394.056091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13399.528124] LustreError: 387290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13399.549332] LustreError: 387290:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13401.452712] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13404.670659] LustreError: 387285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13404.686937] LustreError: 387285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13405.746515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13407.452810] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13410.789668] LustreError: 387285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13410.814696] LustreError: 387285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13414.324862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13414.739942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13419.025040] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13428.459382] Lustre: server umount lustre-OST0000 complete [13434.854314] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13434.867075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13434.884550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13435.373631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13435.391459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13435.406864] Lustre: Skipped 1 previous similar message [13440.481551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13440.486400] Lustre: Skipped 1 previous similar message [13442.614293] Lustre: server umount lustre-MDT0000 complete [13445.607839] LustreError: 389249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13445.635084] LustreError: 389249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13446.559277] Lustre: 386603:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781230737/real 1781230737] req@00000000adbea266 x1867755735246720/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781230753 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13446.596807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13446.944191] LustreError: 387270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230753 with bad export cookie 9780269725760870794 [13446.944974] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13446.957381] LustreError: 387270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13447.642913] Lustre: server umount lustre-MDT0001 complete [13459.029859] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13463.748857] Key type lgssc unregistered [13464.117910] LNet: 390273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13464.122614] LNetError: 390273:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13464.147148] LNet: Removed LNI 192.168.202.136@tcp [13465.390102] Key type .llcrypt unregistered [13465.397197] Key type ._llcrypt unregistered [13494.835559] Key type ._llcrypt registered [13494.837469] Key type .llcrypt registered [13494.936784] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13496.738623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13496.864953] alg: No test for adler32 (adler32-zlib) [13498.012310] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [13498.316668] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13499.967193] Key type lgssc registered [13501.323303] Lustre: Echo OBD driver; http://www.lustre.org/ [13509.510225] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13509.548866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13525.279342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13530.399273] LustreError: 391649:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [13557.090089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13563.831552] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13570.329425] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13577.261362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13577.667190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13577.715215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13581.062491] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13582.829214] LustreError: 391674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13582.852616] LustreError: 391674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13587.941639] LustreError: 391675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13588.568987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13588.969946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13592.879594] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13598.472871] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13602.243319] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13609.441392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13609.455121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13609.474099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13611.522212] Lustre: server umount lustre-OST0000 complete [13617.128592] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13617.138735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13617.152498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13622.753362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13622.761962] Lustre: Skipped 2 previous similar messages [13625.897841] Lustre: server umount lustre-MDT0000 complete [13627.875675] LustreError: 392576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13627.888032] LustreError: 392576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13629.600880] LustreError: 391657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230936 with bad export cookie 17261267851875709915 [13629.620527] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13629.936299] Lustre: server umount lustre-MDT0001 complete [13644.793580] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13649.743901] Key type lgssc unregistered [13650.443774] LNet: 394861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13650.462168] LNetError: 394861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13650.488203] LNet: Removed LNI 192.168.202.136@tcp [13651.630423] Key type .llcrypt unregistered [13651.634247] Key type ._llcrypt unregistered [13677.037742] Key type ._llcrypt registered [13677.039414] Key type .llcrypt registered [13677.104830] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13678.040456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13678.104653] alg: No test for adler32 (adler32-zlib) [13679.284790] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [13679.489817] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13681.135163] Key type lgssc registered [13682.240608] Lustre: Echo OBD driver; http://www.lustre.org/ [13689.465590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13689.497693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13690.989323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13694.625368] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13700.592234] LustreError: 396263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13700.600539] LustreError: 396263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13702.615961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13703.025435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13708.417706] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13715.561288] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13719.867145] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13727.663625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13728.038669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13733.705844] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13735.220927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13738.955918] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13744.108543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13744.118561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13744.143658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13745.633057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13745.635302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13745.643878] Lustre: Skipped 1 previous similar message [13745.648675] Lustre: Skipped 1 previous similar message [13747.185182] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [13747.205584] Lustre: Skipped 1 previous similar message [13749.836701] Lustre: server umount lustre-MDT0000 complete [13752.011071] LustreError: 396243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.36@tcp arrived at 1781231058 with bad export cookie 9075300980297688969 [13752.806334] LustreError: 396986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13752.826400] LustreError: 396986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13753.917441] LustreError: 396244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231060 with bad export cookie 9075300980297688080 [13753.927897] LustreError: 396244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13753.932954] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13757.925627] LustreError: 396263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13757.940176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13757.949045] LustreError: 396263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13757.976101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13757.983953] Lustre: Skipped 3 previous similar messages [13764.065413] LustreError: 396263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13764.067037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13764.093405] LustreError: 396263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13764.109608] Lustre: Skipped 1 previous similar message [13768.159615] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13768.516519] Lustre: server umount lustre-OST0000 complete [13769.189917] LustreError: 396986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13769.236086] LustreError: 396986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13776.440763] LustreError: 396975:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c5cf0b33 x1867756062410496/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13776.686707] Lustre: server umount lustre-MDT0001 complete [13789.805401] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13792.419360] Key type lgssc unregistered [13792.770596] LNet: 399593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13792.777847] LNetError: 399593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13792.793339] LNet: Removed LNI 192.168.202.136@tcp [13793.617676] Key type .llcrypt unregistered [13793.622530] Key type ._llcrypt unregistered [13819.117421] Key type ._llcrypt registered [13819.119795] Key type .llcrypt registered [13819.237989] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13820.274524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13820.383571] alg: No test for adler32 (adler32-zlib) [13821.579108] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [13821.797187] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13823.471199] Key type lgssc registered [13824.638988] Lustre: Echo OBD driver; http://www.lustre.org/ [13833.323943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13833.341601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13834.752585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13838.677144] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13844.986951] LustreError: 400995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13845.011275] LustreError: 400995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13846.752482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13847.170053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13851.207465] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13857.559864] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13862.017682] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13868.915832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13869.238431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13875.974785] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13879.548448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13882.599607] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13888.486092] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13888.505257] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13888.536844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13889.505480] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [13890.027663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13892.704959] Lustre: server umount lustre-MDT0001 complete [13894.601289] LustreError: 400991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13900.269332] LustreError: 400991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13900.282351] LustreError: 400991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13905.376955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13905.387304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13905.392524] LustreError: 400996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13905.410149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13905.428708] Lustre: Skipped 2 previous similar messages [13907.606648] Lustre: server umount lustre-OST0000 complete [13910.508696] LustreError: 402107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13911.537356] Lustre: server umount lustre-MDT0000 complete [13927.781527] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13930.933729] Key type lgssc unregistered [13931.163314] LNet: 404323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13931.184447] LNetError: 404323:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13931.215113] LNet: Removed LNI 192.168.202.136@tcp [13932.231591] Key type .llcrypt unregistered [13932.233346] Key type ._llcrypt unregistered [13963.510775] Key type ._llcrypt registered [13963.512059] Key type .llcrypt registered [13963.636261] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13977.956702] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13978.738151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13978.859185] alg: No test for adler32 (adler32-zlib) [13980.012151] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [13980.203940] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13981.863198] Key type lgssc registered [13982.908694] Lustre: Echo OBD driver; http://www.lustre.org/ [14021.499098] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [14037.433539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14037.478227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14039.096277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14039.157844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14039.326166] Lustre: lustre-MDT0000: new disk, initializing [14039.465239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14039.488676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14044.505876] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14061.748199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14062.151624] Lustre: lustre-OST0000: new disk, initializing [14062.157725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14062.162636] Lustre: Skipped 1 previous similar message [14062.179966] Lustre: 409288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14062.306800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14069.247984] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14069.374044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14069.382336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14069.425795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14077.927631] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14095.439801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14095.558948] Lustre: 408287:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14095.627457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14095.718726] Lustre: lustre-MDT0001: new disk, initializing [14095.891143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14095.942090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14095.955682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14100.593618] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14110.379601] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14110.666950] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14114.610863] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14114.864939] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14121.453789] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [14122.465273] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14122.468764] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14122.486156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14122.491420] Lustre: Skipped 1 previous similar message [14126.031310] Lustre: server umount lustre-MDT0001 complete [14132.709821] LustreError: 408294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14132.728796] LustreError: 408294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [14137.826553] LustreError: 411858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14137.855071] LustreError: 411858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14142.978721] LustreError: 411858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14142.988818] LustreError: 411858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14146.018070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14146.025929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14146.049505] Lustre: Skipped 1 previous similar message [14146.057258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14148.065805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14148.070204] LustreError: 408299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14150.512226] Lustre: server umount lustre-OST0000 complete [14153.185184] LustreError: 408298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14161.599598] Lustre: server umount lustre-MDT0000 complete [14177.613261] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14180.324374] Key type lgssc unregistered [14180.674169] LNet: 412855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14180.685630] LNetError: 412855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14181.734278] LNet: Removed LNI 192.168.202.136@tcp [14182.933039] Key type .llcrypt unregistered [14182.936511] Key type ._llcrypt unregistered [14232.603713] Key type ._llcrypt registered [14232.605642] Key type .llcrypt registered [14232.726645] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [14233.039271] Key type .llcrypt unregistered [14233.041851] Key type ._llcrypt unregistered [14254.464377] Key type ._llcrypt registered [14254.466027] Key type .llcrypt registered [14254.548470] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [14255.794109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14255.848377] alg: No test for adler32 (adler32-zlib) [14256.994330] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [14257.368821] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [14259.111228] Key type lgssc registered [14260.520701] Lustre: Echo OBD driver; http://www.lustre.org/ [14291.470754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14291.548213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14293.001950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14293.047949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14293.170707] Lustre: lustre-MDT0000: new disk, initializing [14293.275517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14293.303039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14298.210600] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14311.640720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14311.938205] Lustre: 416202:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14312.049066] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14312.055610] Lustre: Skipped 1 previous similar message [14312.256720] Lustre: lustre-MDT0001: new disk, initializing [14312.383867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14312.466248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14312.488426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14318.763790] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14326.595352] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14330.704104] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14338.120610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14338.492227] Lustre: lustre-OST0000: new disk, initializing [14338.504252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14338.516177] Lustre: 417946:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14338.625687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14345.952117] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14347.818852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14347.834533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14347.959248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14355.050400] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14363.616509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14363.621618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14363.630582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14363.686229] Lustre: Skipped 2 previous similar messages [14368.740328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14368.764413] Lustre: Skipped 4 previous similar messages [14370.114237] Lustre: server umount lustre-MDT0000 complete [14374.993507] LustreError: 417945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231681 with bad export cookie 5499571800491085356 [14375.015086] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14375.387659] Lustre: server umount lustre-MDT0001 complete [14390.692935] Lustre: server umount lustre-OST0000 complete [14403.537295] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:35:08 (1781231708) [14411.888470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14412.422014] LustreError: 419862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14412.453307] LustreError: 419862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14412.569854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14417.617233] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14417.916544] LustreError: 419863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14423.010230] LustreError: 419862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14425.501047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14425.823658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14429.893455] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14435.408818] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14440.237483] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14460.786217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14461.012442] Lustre: Found index 0 for lustre-OST0000, updating log [14461.177176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14467.626856] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14474.889455] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14483.700097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14492.130117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14492.149975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14492.165534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14498.783501] Lustre: server umount lustre-OST0000 complete [14518.241941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14518.254972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14518.268867] Lustre: Skipped 1 previous similar message [14518.279287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14518.284940] Lustre: Skipped 3 previous similar messages [14521.427242] Lustre: server umount lustre-MDT0000 complete [14521.570068] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [14525.408056] LustreError: 419858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14525.454524] LustreError: 419858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14525.589491] LustreError: 419845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781231832 with bad export cookie 5499571800491086742 [14525.597936] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14525.619304] LustreError: 419845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [14525.933911] Lustre: server umount lustre-MDT0001 complete [14535.677577] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:37:20 (1781231840) [15164.098731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15164.591225] LustreError: 425954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15164.648648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15169.769821] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15170.017846] LustreError: 425955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15175.139013] LustreError: 425954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15180.840748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15181.382927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15186.543333] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15192.724866] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15197.461667] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15216.952105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15217.277353] Lustre: Found index 0 for lustre-OST0000, updating log [15217.606337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15224.043834] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15225.055214] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [15230.115379] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15231.984902] Lustre: server umount lustre-OST0000 complete [15236.833884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15236.851107] Lustre: Skipped 1 previous similar message [15236.858782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15236.866784] Lustre: Skipped 5 previous similar messages [15237.922962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15237.935908] Lustre: Skipped 1 previous similar message [15238.953768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15238.965078] Lustre: Skipped 1 previous similar message [15240.997101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15241.006148] Lustre: Skipped 3 previous similar messages [15245.090978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15245.095463] Lustre: Skipped 6 previous similar messages [15250.401540] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15250.752564] Lustre: server umount lustre-MDT0000 complete [15251.235243] LustreError: 425955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15251.255032] LustreError: 425955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15254.573749] LustreError: 425936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232561 with bad export cookie 5499571800491087834 [15254.581496] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15254.921621] Lustre: server umount lustre-MDT0001 complete [15283.584993] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [15289.568162] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [15291.283508] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15324.811226] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:50:30 (1781232630) [15343.830412] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15346.629601] Key type lgssc unregistered [15346.965732] LNet: 432175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15346.973335] LNetError: 432175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15471.974766] LNet: Removed LNI 192.168.202.136@tcp [15472.799153] Key type .llcrypt unregistered [15472.800630] Key type ._llcrypt unregistered [15489.499066] Key type ._llcrypt registered [15489.500634] Key type .llcrypt registered [15489.620586] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15490.707884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15490.732706] alg: No test for adler32 (adler32-zlib) [15491.844162] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [15492.054276] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15493.719233] Key type lgssc registered [15494.980606] Lustre: Echo OBD driver; http://www.lustre.org/ [15499.576798] LNet: Added LNI 192.168.202.136@tcp100 [8/256/0/180] [15500.510724] LNet: Added LNI 192.168.202.136@tcp101 [8/256/0/180] [15501.773289] LNet: Added LNI 192.168.202.136@tcp102 [8/256/0/180] [15502.941102] LNet: Added LNI 192.168.202.136@tcp103 [8/256/0/180] [15503.789943] LNet: Added LNI 192.168.202.136@tcp104 [8/256/0/180] [15504.738177] LNet: Added LNI 192.168.202.136@tcp105 [8/256/0/180] [15505.598234] LNet: Added LNI 192.168.202.136@tcp106 [8/256/0/180] [15506.418504] LNet: Added LNI 192.168.202.136@tcp107 [8/256/0/180] [15507.354792] LNet: Added LNI 192.168.202.136@tcp108 [8/256/0/180] [15508.158443] LNet: Added LNI 192.168.202.136@tcp109 [8/256/0/180] [15509.048023] LNet: Added LNI 192.168.202.136@tcp110 [8/256/0/180] [15509.868697] LNet: Added LNI 192.168.202.136@tcp111 [8/256/0/180] [15510.878392] LNet: Added LNI 192.168.202.136@tcp112 [8/256/0/180] [15511.712575] LNet: Added LNI 192.168.202.136@tcp113 [8/256/0/180] [15512.758434] LNet: Added LNI 192.168.202.136@tcp114 [8/256/0/180] [15513.572951] LNet: Added LNI 192.168.202.136@tcp115 [8/256/0/180] [15514.301465] LNet: Added LNI 192.168.202.136@tcp116 [8/256/0/180] [15515.176637] LNet: Added LNI 192.168.202.136@tcp117 [8/256/0/180] [15515.966917] LNet: Added LNI 192.168.202.136@tcp118 [8/256/0/180] [15516.780912] LNet: Added LNI 192.168.202.136@tcp119 [8/256/0/180] [15517.624308] LNet: Added LNI 192.168.202.136@tcp120 [8/256/0/180] [15518.546053] LNet: Added LNI 192.168.202.136@tcp121 [8/256/0/180] [15519.729543] LNet: Added LNI 192.168.202.136@tcp122 [8/256/0/180] [15520.659546] LNet: Added LNI 192.168.202.136@tcp123 [8/256/0/180] [15521.588235] LNet: Added LNI 192.168.202.136@tcp124 [8/256/0/180] [15522.475643] LNet: Added LNI 192.168.202.136@tcp125 [8/256/0/180] [15523.407814] LNet: Added LNI 192.168.202.136@tcp126 [8/256/0/180] [15524.448310] LNet: Added LNI 192.168.202.136@tcp127 [8/256/0/180] [15525.458730] LNet: Added LNI 192.168.202.136@tcp128 [8/256/0/180] [15526.438096] LNet: Added LNI 192.168.202.136@tcp129 [8/256/0/180] [15527.412880] LNet: Added LNI 192.168.202.136@tcp130 [8/256/0/180] [15528.373309] LNet: Added LNI 192.168.202.136@tcp131 [8/256/0/180] [15529.244555] LNet: Added LNI 192.168.202.136@tcp132 [8/256/0/180] [15530.129740] LNet: Added LNI 192.168.202.136@tcp133 [8/256/0/180] [15530.908724] LNet: Added LNI 192.168.202.136@tcp134 [8/256/0/180] [15532.182226] LNet: Added LNI 192.168.202.136@tcp135 [8/256/0/180] [15539.502256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15539.527805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15540.716830] Lustre: 435193:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15540.789715] LNetError: 435214:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15540.810112] LNetError: Protocol error connecting to 192.168.202.136@tcp116 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15540.831266] LNetError: Skipped 5 previous similar messages [15540.930668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15540.975822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15541.075149] Lustre: lustre-MDT0000: new disk, initializing [15541.233187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15541.287491] LNet: 8 local NIs in recovery (showing 8): 192.168.202.136@tcp122, 192.168.202.136@tcp123, 192.168.202.136@tcp125, 192.168.202.136@tcp126, 192.168.202.136@tcp127, 192.168.202.136@tcp124, 192.168.202.136@tcp129, 192.168.202.136@tcp128 [15541.293495] LNetError: 432652:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15541.308590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15541.315564] LNetError: 432652:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [15541.315949] LNetError: Protocol error connecting to 192.168.202.136@tcp128 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15541.315952] LNetError: Skipped 8 previous similar messages [15543.392268] LNetError: 432653:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15543.400247] LNetError: Protocol error connecting to 192.168.202.136@tcp124 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15543.413549] LNetError: Skipped 2 previous similar messages [15546.773660] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15547.506732] LNetError: 435213:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15547.527237] LNetError: 435213:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15547.538231] LNetError: Protocol error connecting to 192.168.202.136@tcp124 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15555.687387] LNetError: 432651:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15555.710074] LNetError: Protocol error connecting to 192.168.202.136@tcp124 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15557.569723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15557.694412] Lustre: 435210:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15557.730281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15557.734466] Lustre: Skipped 1 previous similar message [15557.812909] Lustre: lustre-MDT0001: new disk, initializing [15557.858599] Lustre: 435997:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15557.867909] Lustre: 435997:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15557.985418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15558.071925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15558.091556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15562.066434] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15565.342456] Lustre: 436362:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15565.355684] Lustre: 436362:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15569.550621] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15570.921797] LNetError: 432652:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15570.930087] LNetError: Protocol error connecting to 192.168.202.136@tcp104 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15573.783831] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15580.405705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15580.833246] Lustre: lustre-OST0000: new disk, initializing [15580.840052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15580.855400] Lustre: 436963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15580.962620] Lustre: 436968:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15580.971478] Lustre: 436968:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [15581.036133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15586.557739] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15588.320383] LNetError: 435214:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15588.338038] LNetError: 435214:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15588.348950] LNetError: Protocol error connecting to 192.168.202.136@tcp126 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15588.358954] LNetError: Skipped 10 previous similar messages [15588.904071] Lustre: 437402:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15588.917194] Lustre: 437402:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [15589.007556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15589.016673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15589.145759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15593.996356] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15601.759454] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp124 [15604.705569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15604.707318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15604.719408] LustreError: Skipped 1 previous similar message [15604.722516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15604.734453] Lustre: Skipped 1 previous similar message [15607.465396] Lustre: server umount lustre-OST0000 complete [15614.437076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15614.450856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15614.460136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15614.464455] Lustre: Skipped 1 previous similar message [15618.453263] Lustre: server umount lustre-MDT0000 complete [15621.091237] LNetError: 435220:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15621.108327] LNetError: 435220:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [15621.120286] LNetError: Protocol error connecting to 192.168.202.136@tcp104 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15621.129651] LNetError: Skipped 20 previous similar messages [15622.362635] LustreError: 435201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232929 with bad export cookie 17894105330243740580 [15622.374449] LustreError: 435201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15622.375807] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15622.954560] Lustre: server umount lustre-MDT0001 complete [15633.705708] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15636.537955] Key type lgssc unregistered [15636.883223] LNet: 438744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15636.891287] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15637.920195] LNet: Removed LNI 192.168.202.136@tcp [15637.922674] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15637.942173] LNet: Removed LNI 192.168.202.136@tcp100 [15639.007212] LNet: Removed LNI 192.168.202.136@tcp101 [15639.009389] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15639.020884] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15639.036673] LNet: Removed LNI 192.168.202.136@tcp102 [15640.095161] LNet: Removed LNI 192.168.202.136@tcp103 [15641.121930] LNet: Removed LNI 192.168.202.136@tcp104 [15641.124172] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15641.139204] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15641.157485] LNet: Removed LNI 192.168.202.136@tcp105 [15642.207272] LNet: Removed LNI 192.168.202.136@tcp106 [15643.231267] LNet: Removed LNI 192.168.202.136@tcp107 [15644.257990] LNet: Removed LNI 192.168.202.136@tcp108 [15645.279620] LNet: Removed LNI 192.168.202.136@tcp109 [15645.283862] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15645.293729] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15645.310885] LNet: Removed LNI 192.168.202.136@tcp110 [15646.367172] LNet: Removed LNI 192.168.202.136@tcp111 [15647.391568] LNet: Removed LNI 192.168.202.136@tcp112 [15648.415212] LNet: Removed LNI 192.168.202.136@tcp113 [15649.439192] LNet: Removed LNI 192.168.202.136@tcp114 [15650.463171] LNet: Removed LNI 192.168.202.136@tcp115 [15651.487128] LNet: Removed LNI 192.168.202.136@tcp116 [15652.576605] LNet: Removed LNI 192.168.202.136@tcp117 [15653.599180] LNet: Removed LNI 192.168.202.136@tcp118 [15653.602182] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15653.617426] LNetError: 438744:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15653.637356] LNet: Removed LNI 192.168.202.136@tcp119 [15654.687241] LNet: Removed LNI 192.168.202.136@tcp120 [15655.711697] LNet: Removed LNI 192.168.202.136@tcp121 [15656.735639] LNet: Removed LNI 192.168.202.136@tcp122 [15657.825596] LNet: Removed LNI 192.168.202.136@tcp123 [15657.831121] LNet: Removed LNI 192.168.202.136@tcp124 [15658.911181] LNet: Removed LNI 192.168.202.136@tcp125 [15659.936504] LNet: Removed LNI 192.168.202.136@tcp126 [15660.960707] LNet: Removed LNI 192.168.202.136@tcp127 [15661.983201] LNet: Removed LNI 192.168.202.136@tcp128 [15663.008471] LNet: Removed LNI 192.168.202.136@tcp129 [15663.010103] LNet: Removed LNI 192.168.202.136@tcp130 [15663.012985] LNet: Removed LNI 192.168.202.136@tcp131 [15663.019663] LNet: Removed LNI 192.168.202.136@tcp132 [15663.023961] LNet: Removed LNI 192.168.202.136@tcp133 [15663.030688] LNet: Removed LNI 192.168.202.136@tcp134 [15663.054421] LNet: Removed LNI 192.168.202.136@tcp135 [15663.936824] Key type .llcrypt unregistered [15663.945423] Key type ._llcrypt unregistered [15689.385639] Key type ._llcrypt registered [15689.388476] Key type .llcrypt registered [15689.528031] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [15703.722581] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15705.000254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15705.033171] alg: No test for adler32 (adler32-zlib) [15706.144416] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [15706.459830] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15708.191455] Key type lgssc registered [15709.768223] Lustre: Echo OBD driver; http://www.lustre.org/ [15746.494040] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:57:31 (1781233051) [15764.911583] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15767.301954] Key type lgssc unregistered [15767.584680] LNet: 442782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15767.596398] LNetError: 442782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15767.617945] LNet: Removed LNI 192.168.202.136@tcp [15768.502490] Key type .llcrypt unregistered [15768.507097] Key type ._llcrypt unregistered [15785.505356] Key type ._llcrypt registered [15785.506644] Key type .llcrypt registered [15785.611082] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15786.621209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15786.732795] alg: No test for adler32 (adler32-zlib) [15787.814568] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [15787.967442] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15789.602440] Key type lgssc registered [15790.400326] Lustre: Echo OBD driver; http://www.lustre.org/ [15797.998909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15798.013428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15799.309091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15799.348212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15799.440692] Lustre: lustre-MDT0000: new disk, initializing [15799.569421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15799.592474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15804.646306] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15816.181601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15816.335832] Lustre: 443978:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15816.391655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15816.395771] Lustre: Skipped 1 previous similar message [15816.488691] Lustre: lustre-MDT0001: new disk, initializing [15816.620547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15816.671037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15816.687553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15822.306367] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15831.989841] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15835.728518] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15842.071834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15842.352497] Lustre: lustre-OST0000: new disk, initializing [15842.357787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15842.393089] Lustre: 445721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15842.505970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15848.006704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15848.022149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15848.134826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15849.355395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15856.704609] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15860.493838] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing lnet_if_list [15866.000543] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15866.024732] LNet: Added LNI 192.168.202.136@tcp10 [8/256/0/180] [15870.371682] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15870.395638] LNet: Added LNI 192.168.202.136@tcp11 [8/256/0/180] [15874.479293] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15874.506584] LNet: Added LNI 192.168.202.136@tcp12 [8/256/0/180] [15878.482170] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15878.526867] LNet: Added LNI 192.168.202.136@tcp13 [8/256/0/180] [15882.712880] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15882.735899] LNet: Added LNI 192.168.202.136@tcp14 [8/256/0/180] [15887.153269] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15887.189850] LNet: Added LNI 192.168.202.136@tcp15 [8/256/0/180] [15891.636495] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15891.659112] LNet: Added LNI 192.168.202.136@tcp16 [8/256/0/180] [15896.598076] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15896.656593] LNet: Added LNI 192.168.202.136@tcp17 [8/256/0/180] [15902.418212] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15902.466562] LNet: Added LNI 192.168.202.136@tcp18 [8/256/0/180] [15908.229584] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15908.271450] LNet: Added LNI 192.168.202.136@tcp19 [8/256/0/180] [15913.804454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15913.844642] LNet: Added LNI 192.168.202.136@tcp20 [8/256/0/180] [15919.830419] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15919.859149] LNet: Added LNI 192.168.202.136@tcp21 [8/256/0/180] [15925.322372] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15925.347755] LNet: Added LNI 192.168.202.136@tcp22 [8/256/0/180] [15931.062158] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15931.091035] LNet: Added LNI 192.168.202.136@tcp23 [8/256/0/180] [15937.339902] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15937.380166] LNet: Added LNI 192.168.202.136@tcp24 [8/256/0/180] [15943.645723] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15943.693753] LNet: Added LNI 192.168.202.136@tcp25 [8/256/0/180] [15949.199271] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15949.219557] LNet: Added LNI 192.168.202.136@tcp26 [8/256/0/180] [15954.264803] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15954.296192] LNet: Added LNI 192.168.202.136@tcp27 [8/256/0/180] [15961.001434] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15961.050984] LNet: Added LNI 192.168.202.136@tcp28 [8/256/0/180] [15967.357412] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15967.400064] LNet: Added LNI 192.168.202.136@tcp29 [8/256/0/180] [15973.701894] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15973.734911] LNet: Added LNI 192.168.202.136@tcp30 [8/256/0/180] [15979.857438] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15979.883530] LNet: Added LNI 192.168.202.136@tcp31 [8/256/0/180] [15986.432455] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15986.460836] LNet: Added LNI 192.168.202.136@tcp32 [8/256/0/180] [15993.979440] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15994.049468] LNet: Added LNI 192.168.202.136@tcp33 [8/256/0/180] [16002.745191] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16002.773198] LNet: Added LNI 192.168.202.136@tcp34 [8/256/0/180] [16010.061125] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16010.097095] LNet: Added LNI 192.168.202.136@tcp35 [8/256/0/180] [16018.196831] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16018.252997] LNet: Added LNI 192.168.202.136@tcp36 [8/256/0/180] [16027.070458] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16027.104966] LNet: Added LNI 192.168.202.136@tcp37 [8/256/0/180] [16034.545344] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16034.576606] LNet: Added LNI 192.168.202.136@tcp38 [8/256/0/180] [16041.554685] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16041.581767] LNet: Added LNI 192.168.202.136@tcp39 [8/256/0/180] [16048.969951] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16048.998962] LNet: Added LNI 192.168.202.136@tcp40 [8/256/0/180] [16055.855847] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16055.889451] LNet: Added LNI 192.168.202.136@tcp41 [8/256/0/180] [16063.029290] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16063.060236] LNet: Added LNI 192.168.202.136@tcp42 [8/256/0/180] [16070.671194] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16070.696196] LNet: Added LNI 192.168.202.136@tcp43 [8/256/0/180] [16078.380389] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16078.405247] LNet: Added LNI 192.168.202.136@tcp44 [8/256/0/180] [16085.958870] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16085.986884] LNet: Added LNI 192.168.202.136@tcp45 [8/256/0/180] [16093.265724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16093.302967] LNet: Added LNI 192.168.202.136@tcp46 [8/256/0/180] [16100.380801] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16100.413623] LNet: Added LNI 192.168.202.136@tcp47 [8/256/0/180] [16109.235983] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16109.260830] LNet: Added LNI 192.168.202.136@tcp48 [8/256/0/180] [16119.668642] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16119.700084] LNet: Added LNI 192.168.202.136@tcp49 [8/256/0/180] [16129.763722] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16129.792053] LNet: Added LNI 192.168.202.136@tcp50 [8/256/0/180] [16150.498271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16150.502174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16150.521246] Lustre: Skipped 2 previous similar messages [16150.533313] Lustre: Skipped 2 previous similar messages [16155.630508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16155.638629] Lustre: Skipped 2 previous similar messages [16156.673364] Lustre: server umount lustre-MDT0000 complete [16160.616482] LustreError: 443970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233467 with bad export cookie 14992589772232299044 [16160.633400] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16160.742393] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16160.885740] Lustre: server umount lustre-MDT0001 complete [16182.105517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16192.324496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16206.153827] LustreError: 445715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16206.188856] LustreError: 445715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [16206.289250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16210.624352] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16211.433267] LustreError: 445716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16215.534161] LustreError: 445715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16215.548167] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16218.595135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16219.006977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16223.893987] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16225.279138] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16230.794995] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16235.620667] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16257.504308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16257.519575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16257.538924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16260.065309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16260.074177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16260.084236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16262.766351] Lustre: server umount lustre-OST0000 complete [16270.305295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16270.314768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16270.353836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16270.362217] Lustre: Skipped 2 previous similar messages [16273.554085] Lustre: server umount lustre-MDT0000 complete [16276.453727] LustreError: 469132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16276.479879] LustreError: 469132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16278.050282] LustreError: 444767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233584 with bad export cookie 14992589772232301690 [16278.052210] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16278.061474] LustreError: 444767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [16278.482715] Lustre: server umount lustre-MDT0001 complete [16288.815946] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16291.541743] Key type lgssc unregistered [16292.034300] LNet: 471683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16292.068477] LNetError: 471683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16292.118847] LNet: Removed LNI 192.168.202.136@tcp [16292.124053] LNet: Removed LNI 192.168.202.136@tcp10 [16292.139117] LNet: Removed LNI 192.168.202.136@tcp11 [16292.141094] LNet: Removed LNI 192.168.202.136@tcp12 [16292.155555] LNet: Removed LNI 192.168.202.136@tcp13 [16292.160936] LNet: Removed LNI 192.168.202.136@tcp14 [16292.172969] LNet: Removed LNI 192.168.202.136@tcp15 [16292.177750] LNet: Removed LNI 192.168.202.136@tcp16 [16292.185562] LNet: Removed LNI 192.168.202.136@tcp17 [16292.193902] LNet: Removed LNI 192.168.202.136@tcp18 [16292.195867] LNet: Removed LNI 192.168.202.136@tcp19 [16292.218601] LNet: Removed LNI 192.168.202.136@tcp20 [16292.229617] LNet: Removed LNI 192.168.202.136@tcp21 [16292.236196] LNet: Removed LNI 192.168.202.136@tcp22 [16292.254886] LNet: Removed LNI 192.168.202.136@tcp23 [16292.265426] LNet: Removed LNI 192.168.202.136@tcp24 [16292.288926] LNet: Removed LNI 192.168.202.136@tcp25 [16292.290950] LNet: Removed LNI 192.168.202.136@tcp26 [16292.314590] LNet: Removed LNI 192.168.202.136@tcp27 [16292.326455] LNet: Removed LNI 192.168.202.136@tcp28 [16292.338506] LNet: Removed LNI 192.168.202.136@tcp29 [16292.349242] LNet: Removed LNI 192.168.202.136@tcp30 [16292.351227] LNet: Removed LNI 192.168.202.136@tcp31 [16292.358325] LNet: Removed LNI 192.168.202.136@tcp32 [16292.360200] LNet: Removed LNI 192.168.202.136@tcp33 [16292.370617] LNet: Removed LNI 192.168.202.136@tcp34 [16292.382988] LNet: Removed LNI 192.168.202.136@tcp35 [16292.391711] LNet: Removed LNI 192.168.202.136@tcp36 [16292.400324] LNet: Removed LNI 192.168.202.136@tcp37 [16292.402368] LNet: Removed LNI 192.168.202.136@tcp38 [16292.414633] LNet: Removed LNI 192.168.202.136@tcp39 [16292.417690] LNet: Removed LNI 192.168.202.136@tcp40 [16292.427587] LNet: Removed LNI 192.168.202.136@tcp41 [16292.439928] LNet: Removed LNI 192.168.202.136@tcp42 [16292.461489] LNet: Removed LNI 192.168.202.136@tcp43 [16292.463487] LNet: Removed LNI 192.168.202.136@tcp44 [16292.474736] LNet: Removed LNI 192.168.202.136@tcp45 [16292.482691] LNet: Removed LNI 192.168.202.136@tcp46 [16292.499662] LNet: Removed LNI 192.168.202.136@tcp47 [16292.509218] LNet: Removed LNI 192.168.202.136@tcp48 [16292.521482] LNet: Removed LNI 192.168.202.136@tcp49 [16292.543806] LNet: Removed LNI 192.168.202.136@tcp50 [16293.565995] Key type .llcrypt unregistered [16293.570491] Key type ._llcrypt unregistered [16317.698882] Key type ._llcrypt registered [16317.706732] Key type .llcrypt registered [16317.793379] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16334.842640] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16335.788231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16335.994958] alg: No test for adler32 (adler32-zlib) [16337.038545] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [16337.196367] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16338.832603] Key type lgssc registered [16340.239215] Lustre: Echo OBD driver; http://www.lustre.org/ [16382.050117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16382.071653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16383.459754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16383.531231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16383.679360] Lustre: lustre-MDT0000: new disk, initializing [16383.787717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16383.821752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16389.455970] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16400.612247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16400.745771] Lustre: 474848:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16400.805994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16400.810619] Lustre: Skipped 1 previous similar message [16400.920763] Lustre: lustre-MDT0001: new disk, initializing [16401.026639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16401.062676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16401.070511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16405.025988] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16412.857896] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16416.419331] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16423.965782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16424.182328] Lustre: *** cfs_fail_loc=90f, val=0*** [16424.276681] Lustre: lustre-OST0000: new disk, initializing [16424.282566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16424.296560] Lustre: 476637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16424.380235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16430.474757] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16431.697147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16431.710255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16431.800300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16439.267607] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16452.576950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16452.578571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16452.593122] LustreError: Skipped 1 previous similar message [16452.598481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16452.619170] Lustre: Skipped 1 previous similar message [16452.667936] Lustre: Skipped 1 previous similar message [16454.516905] Lustre: server umount lustre-OST0000 complete [16462.838291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16462.840210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16462.863194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16462.867890] Lustre: Skipped 1 previous similar message [16462.877965] Lustre: Skipped 1 previous similar message [16465.043264] Lustre: server umount lustre-MDT0000 complete [16470.143676] LustreError: 474841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233777 with bad export cookie 10734766455176912712 [16470.150664] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16470.155250] LustreError: 474841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16470.955425] Lustre: server umount lustre-MDT0001 complete [16485.520287] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16488.953797] Key type lgssc unregistered [16489.463375] LNet: 478414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16489.475577] LNetError: 478414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16489.492894] LNet: Removed LNI 192.168.202.136@tcp [16490.576888] Key type .llcrypt unregistered [16490.578965] Key type ._llcrypt unregistered [16514.643483] Key type ._llcrypt registered [16514.645614] Key type .llcrypt registered [16514.749284] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16515.917127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16516.079653] alg: No test for adler32 (adler32-zlib) [16517.215971] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [16517.424698] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16519.095885] Key type lgssc registered [16520.652830] Lustre: Echo OBD driver; http://www.lustre.org/ [16528.972800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16529.002587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16530.636874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16535.936481] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16540.131496] LustreError: 479667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16540.162935] LustreError: 479667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16545.262950] LustreError: 479668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16545.517575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16545.981986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16550.368623] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16556.644785] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16561.016875] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16569.049552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16569.433034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16575.748295] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16581.956584] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16594.380097] loop0: detected capacity change from 0 to 204800000 [16594.686606] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16594.718387] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16594.747643] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16601.487841] loop0: detected capacity change from 0 to 204800000 [16601.779330] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16601.791364] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16601.805947] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16607.223394] loop0: detected capacity change from 0 to 204800000 [16613.746224] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16613.881139] Lustre: fs15246-OST0000: new disk, initializing [16613.887397] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16613.894484] Lustre: 482925:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16613.942752] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16620.072670] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16630.367110] loop1: detected capacity change from 0 to 204800000 [16636.184869] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16636.313371] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16636.389605] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16636.476753] Lustre: fs15246-MDT0000: new disk, initializing [16636.653427] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16636.687369] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16636.797892] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16637.857424] Lustre: 484031:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000095b034a x1867759038280704/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781233960 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16641.833796] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16646.963953] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16646.971918] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16647.028596] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16670.695384] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16670.711378] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16673.652900] Lustre: server umount fs15246-MDT0000 complete [16675.820717] LustreError: 480889:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16680.967976] LustreError: 481773:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16686.064594] LustreError: 480889:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16686.603842] Lustre: server umount fs15246-OST0000 complete [16701.410700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16701.420824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16701.422100] Lustre: Skipped 1 previous similar message [16701.437452] Lustre: Skipped 1 previous similar message [16706.537283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16706.540269] Lustre: Skipped 1 previous similar message [16711.652144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16711.662263] Lustre: Skipped 1 previous similar message [16714.207554] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16714.398893] Lustre: server umount lustre-OST0000 complete [16719.841374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16719.849472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16719.862994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16721.895423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16725.544917] Lustre: server umount lustre-MDT0000 complete [16727.011441] LustreError: 479668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16727.028746] LustreError: 479668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16729.698598] LustreError: 479648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234036 with bad export cookie 3459150227304359058 [16729.704033] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16729.709207] LustreError: 479648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16730.086702] Lustre: server umount lustre-MDT0001 complete [16741.595868] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16744.526905] Key type lgssc unregistered [16744.820875] LNet: 487095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16744.836693] LNetError: 487095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16744.857231] LNet: Removed LNI 192.168.202.136@tcp [16745.658170] Key type .llcrypt unregistered [16745.660217] Key type ._llcrypt unregistered [16771.804144] Key type ._llcrypt registered [16771.806595] Key type .llcrypt registered [16771.879727] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16772.816911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16772.869570] alg: No test for adler32 (adler32-zlib) [16773.859901] Lustre: Lustre: Build Version: 2.17.53_75_g2ada427 [16774.133059] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16775.815172] Key type lgssc registered [16777.363641] Lustre: Echo OBD driver; http://www.lustre.org/ [16817.511684] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16827.621454] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16869.666433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16869.715652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16871.006187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16871.033148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16871.119953] Lustre: lustre-MDT0000: new disk, initializing [16871.194403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16871.209987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16875.557272] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16887.211194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16887.350561] Lustre: 491514:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16887.384914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16887.387072] Lustre: Skipped 1 previous similar message [16887.438763] Lustre: lustre-MDT0001: new disk, initializing [16887.523599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16887.563976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16887.578058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16892.114614] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16900.371174] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16904.178240] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16911.649419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16911.931281] Lustre: lustre-OST0000: new disk, initializing [16911.941302] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16911.962862] Lustre: 493257:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16912.072079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16917.556881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16917.570076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16917.741945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16917.811801] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16925.340323] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16930.585209] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16931.011695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16936.088724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16936.274661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16938.465661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16938.478520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16938.479703] LustreError: Skipped 1 previous similar message [16938.493118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16938.513573] Lustre: Skipped 1 previous similar message [16943.593845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16943.609665] Lustre: Skipped 1 previous similar message [16944.919591] Lustre: server umount lustre-OST0000 complete [16953.827938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16953.829387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16953.840557] Lustre: Skipped 1 previous similar message [16953.856844] Lustre: Skipped 2 previous similar messages [16956.011757] Lustre: server umount lustre-MDT0000 complete [16960.279989] LustreError: 491506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234267 with bad export cookie 10267300969323754144 [16960.292335] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16960.297337] LustreError: 491506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16960.632847] Lustre: server umount lustre-MDT0001 complete [16990.107767] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [17001.975497] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [17034.961897] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16694 sec ========= 23:18:59 (1781234339) [17036.739271] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:19:01 (1781234341) === [17040.377632] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:19:05 (1781234345) === [17082.183380] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [17084.813780] Key type lgssc unregistered [17085.037919] LNet: 499702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17085.052335] LNetError: 499702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [17085.061854] LNet: Removed LNI 192.168.202.136@tcp [17085.864280] Key type .llcrypt unregistered [17085.868607] Key type ._llcrypt unregistered