[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474805385 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003119] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008382] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012045] Yama: becoming mindful. [ 0.013028] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.022662] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027530] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.032318] *** VALIDATE proc *** [ 0.033227] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035252] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039017] Speculative Store Bypass: Vulnerable [ 0.042454] debug: unmapping init [mem 0xffffffff8b459000-0xffffffff8b460fff] [ 0.044165] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045663] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046031] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053279] rcu: Hierarchical SRCU implementation. [ 0.055365] smp: Bringing up secondary CPUs ... [ 0.056509] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.060407] smp: Brought up 1 node, 4 CPUs [ 0.062029] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150845] node 0 deferred pages initialised in 84ms [ 0.154121] devtmpfs: initialized [ 0.155268] x86/mm: Memory block size: 128MB [ 0.160832] gcov: version magic: 0x41383552 [ 0.161650] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168199] pinctrl core: initialized pinctrl subsystem [ 0.170265] [ 0.170781] ************************************************************* [ 0.173013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175012] ** ** [ 0.177011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179011] ** ** [ 0.180010] ** This means that this kernel is built to expose internal ** [ 0.182012] ** IOMMU data structures, which may compromise security on ** [ 0.184022] ** your system. ** [ 0.187013] ** ** [ 0.188010] ** If you see this message and you are not debugging the ** [ 0.190011] ** kernel, report this immediately to your vendor! ** [ 0.192011] ** ** [ 0.194010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196009] ************************************************************* [ 0.197742] NET: Registered protocol family 16 [ 0.199366] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209011] cpuidle: using governor menu [ 0.210722] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213514] PCI: Using configuration type 1 for base access [ 0.216125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224053] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.225037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227034] cryptd: max_cpu_qlen set to 1000 [ 0.230101] ACPI: Added _OSI(Module Device) [ 0.231014] ACPI: Added _OSI(Processor Device) [ 0.233017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246463] ACPI: Interpreter enabled [ 0.248110] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250013] ACPI: Using IOAPIC for interrupt routing [ 0.252101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255405] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266013] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277168] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283344] acpiphp: Slot [2] registered [ 0.285133] acpiphp: Slot [5] registered [ 0.286204] acpiphp: Slot [6] registered [ 0.288135] acpiphp: Slot [7] registered [ 0.290137] acpiphp: Slot [8] registered [ 0.292141] acpiphp: Slot [9] registered [ 0.293115] acpiphp: Slot [10] registered [ 0.295172] acpiphp: Slot [3] registered [ 0.296104] acpiphp: Slot [4] registered [ 0.298116] acpiphp: Slot [11] registered [ 0.300169] acpiphp: Slot [12] registered [ 0.302145] acpiphp: Slot [13] registered [ 0.304140] acpiphp: Slot [14] registered [ 0.306132] acpiphp: Slot [15] registered [ 0.307106] acpiphp: Slot [16] registered [ 0.308089] acpiphp: Slot [17] registered [ 0.309078] acpiphp: Slot [18] registered [ 0.310074] acpiphp: Slot [19] registered [ 0.312110] acpiphp: Slot [20] registered [ 0.313107] acpiphp: Slot [21] registered [ 0.315102] acpiphp: Slot [22] registered [ 0.317105] acpiphp: Slot [23] registered [ 0.318094] acpiphp: Slot [24] registered [ 0.320058] acpiphp: Slot [25] registered [ 0.321058] acpiphp: Slot [26] registered [ 0.322089] acpiphp: Slot [27] registered [ 0.323093] acpiphp: Slot [28] registered [ 0.325092] acpiphp: Slot [29] registered [ 0.326116] acpiphp: Slot [30] registered [ 0.328102] acpiphp: Slot [31] registered [ 0.329067] PCI host bridge to bus 0000:00 [ 0.331020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334049] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336062] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346849] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350213] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358628] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.364019] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.370019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373534] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378304] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382778] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.387015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.401016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.405019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.411719] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.420016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.427014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457568] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.508465] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.523017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.551439] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561030] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.585027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593835] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.607017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.614016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.643897] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.651015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.680178] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.683325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.685317] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.687324] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.689271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.694123] iommu: Default domain type: Passthrough [ 0.695391] SCSI subsystem initialized [ 0.697148] ACPI: bus type USB registered [ 0.698098] usbcore: registered new interface driver usbfs [ 0.701082] usbcore: registered new interface driver hub [ 0.703080] usbcore: registered new device driver usb [ 0.705169] pps_core: LinuxPPS API ver. 1 registered [ 0.706007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.709061] PTP clock support registered [ 0.711128] EDAC MC: Ver: 3.0.0 [ 0.713502] PCI: Using ACPI for IRQ routing [ 0.715980] NetLabel: Initializing [ 0.717011] NetLabel: domain hash size = 128 [ 0.718008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.720073] NetLabel: unlabeled traffic allowed by default [ 0.722115] vgaarb: loaded [ 0.723247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.725019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.730994] clocksource: Switched to clocksource kvm-clock [ 0.838592] VFS: Disk quotas dquot_6.6.0 [ 0.840024] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.841848] *** VALIDATE ramfs *** [ 0.843284] *** VALIDATE hugetlbfs *** [ 0.845637] pnp: PnP ACPI init [ 0.848707] pnp: PnP ACPI: found 6 devices [ 0.865198] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.869082] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871692] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.873736] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.876341] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.878681] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.881302] NET: Registered protocol family 2 [ 0.883525] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.887703] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.892414] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.898891] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.902450] TCP: Hash tables configured (established 65536 bind 65536) [ 0.905747] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.909526] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913721] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917301] NET: Registered protocol family 1 [ 0.920394] RPC: Registered named UNIX socket transport module. [ 0.923233] RPC: Registered udp transport module. [ 0.925439] RPC: Registered tcp transport module. [ 0.927477] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.930223] NET: Registered protocol family 44 [ 0.931875] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.934340] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.936434] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.938469] PCI: CLS 0 bytes, default 64 [ 0.940129] Unpacking initramfs... [ 2.365957] debug: unmapping init [mem 0xffff90b4bcc54000-0xffff90b4bffbffff] [ 2.370347] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373065] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.376394] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.932749] Initialise system trusted keyrings [ 2.934884] Key type blacklist registered [ 2.945812] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.956636] zbud: loaded [ 2.959986] *** VALIDATE nfs *** [ 2.961603] *** VALIDATE nfs4 *** [ 2.964170] pstore: using deflate compression [ 2.968792] Platform Keyring initialized [ 3.073494] NET: Registered protocol family 38 [ 3.075855] Key type asymmetric registered [ 3.077609] Asymmetric key parser 'x509' registered [ 3.080050] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.083538] io scheduler mq-deadline registered [ 3.085454] io scheduler kyber registered [ 3.086792] io scheduler bfq registered [ 3.088160] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.090815] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.093056] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.095475] ACPI: Power Button [PWRF] [ 3.100418] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.109686] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.126973] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.138837] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.164382] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.191763] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.221846] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.227701] Non-volatile memory driver v1.3 [ 3.229607] Linux agpgart interface v0.103 [ 3.262929] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.266030] vda: detected capacity change from 0 to 74698752 [ 3.283608] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.287233] vdb: detected capacity change from 0 to 1073741824 [ 3.303156] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306490] vdc: detected capacity change from 0 to 2621440000 [ 3.323168] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326221] vdd: detected capacity change from 0 to 2621440000 [ 3.340585] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.343188] vde: detected capacity change from 0 to 4294967296 [ 3.357785] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.361194] vdf: detected capacity change from 0 to 4294967296 [ 3.367291] libphy: Fixed MDIO Bus: probed [ 3.371947] usbcore: registered new interface driver usbserial_generic [ 3.374779] usbserial: USB Serial support registered for generic [ 3.377703] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.382265] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.384082] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.386633] mousedev: PS/2 mouse device common for all mice [ 3.389425] rtc_cmos 00:05: RTC can wake from S4 [ 3.392260] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.396057] rtc_cmos 00:05: registered as rtc0 [ 3.399213] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.402112] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.405919] intel_pstate: CPU model not supported [ 3.412595] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.418750] hid: raw HID events driver (C) Jiri Kosina [ 3.421213] usbcore: registered new interface driver usbhid [ 3.423704] usbhid: USB HID core driver [ 3.425491] drop_monitor: Initializing network drop monitor service [ 3.428259] Initializing XFRM netlink socket [ 3.430526] NET: Registered protocol family 10 [ 3.434378] Segment Routing with IPv6 [ 3.435819] NET: Registered protocol family 17 [ 3.439062] mpls_gso: MPLS GSO support [ 3.444381] RAS: Correctable Errors collector initialized. [ 3.447855] AVX version of gcm_enc/dec engaged. [ 3.449960] AES CTR mode by8 optimization enabled [ 3.534711] sched_clock: Marking stable (3534686786, 0)->(4472072738, -937385952) [ 3.539260] registered taskstats version 1 [ 3.542217] Loading compiled-in X.509 certificates [ 3.543815] zswap: loaded using pool lzo/zbud [ 3.565207] Key type big_key registered [ 3.577762] Key type encrypted registered [ 3.579465] ima: No TPM chip found, activating TPM-bypass! [ 3.581796] ima: Allocated hash algorithm: sha1 [ 3.584416] ima: No architecture policies found [ 3.587126] evm: Initialising EVM extended attributes: [ 3.590212] evm: security.selinux [ 3.592691] evm: security.ima [ 3.594108] evm: security.capability [ 3.595986] evm: HMAC attrs: 0x1 [ 3.598741] rtc_cmos 00:05: setting system clock to 2026-09-03 13:09:03 UTC (1788440943) [ 3.606900] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 3.610978] debug: unmapping init [mem 0xffffffff8b182000-0xffffffff8b458fff] [ 3.621073] Write protecting the kernel read-only data: 28672k [ 3.625501] debug: unmapping init [mem 0xffffffff89803000-0xffffffff899fffff] [ 3.628370] debug: unmapping init [mem 0xffffffff8a114000-0xffffffff8a1fffff] [ 3.663905] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.674844] systemd[1]: Detected virtualization kvm. [ 3.676890] systemd[1]: Detected architecture x86-64. [ 3.679171] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.703841] systemd[1]: No hostname configured. [ 3.705062] systemd[1]: Set hostname to . [ 3.706749] random: systemd: uninitialized urandom read (16 bytes read) [ 3.708789] systemd[1]: Initializing machine ID from random generator. [ 3.747993] random: ln: uninitialized urandom read (6 bytes read) [ 3.825431] random: systemd: uninitialized urandom read (16 bytes read) [ 3.828152] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.833151] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.843995] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook.[ 4.405748] random: fast init done Starting dracut pre-udev hook... [ 4.467144] device-mapper: uevent: version 1.0.3 [ 4.469792] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.211826] virtio_net virtio0 ens2: renamed from eth0 [ 5.350771] scsi host0: ata_piix [ 5.396298] scsi host1: ata_piix [ 5.397931] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.401121] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.919587] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.102174] random: crng init done [ 10.103683] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.683565] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.883420] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.182096] SELinux: Disabled at runtime. [ 12.250361] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.262083] systemd[1]: Detected virtualization kvm. [ 12.264203] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.790343] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.794518] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.799145] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.804283] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.809190] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.819144] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.823442] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ 12.951954] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.312985] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.768535] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.779930] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.989482] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.005755] EDAC sbridge: Ver: 1.1.2 [ 15.751391] Key type dns_resolver registered [ 16.079633] NFS: Registering the id_resolver key type [ 16.082265] Key type id_resolver registered [ 16.084098] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg335-server login: [ 45.577119] hrtimer: interrupt took 2207863 ns [ 66.668702] libcfs: loading out-of-tree module taints kernel. [ 66.726792] Key type ._llcrypt registered [ 66.733093] Key type .llcrypt registered [ 66.844247] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 85.271561] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 86.943251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.968646] alg: No test for adler32 (adler32-zlib) [ 88.361608] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 89.446364] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 91.183182] Key type lgssc registered [ 93.265985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.768439] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.658872] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 159.950508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.971804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.192913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.211550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.293656] Lustre: lustre-MDT0000: new disk, initializing [ 161.424166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.448293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.221913] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.560781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.699290] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.740646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.749026] Lustre: Skipped 1 previous similar message [ 176.873397] Lustre: lustre-MDT0001: new disk, initializing [ 177.001364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.053934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.078718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.038605] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.782885] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.418723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.637697] Lustre: lustre-OST0000: new disk, initializing [ 199.640577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.644369] Lustre: 8467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.692250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.592250] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.471169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.480099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.563737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.455375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.590564] Lustre: lustre-OST0001: new disk, initializing [ 219.594499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.600234] Lustre: 9539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.660848] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.479788] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.365665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.384311] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.456024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.284396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.306894] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.303574] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing check_logdir /tmp/testlogs/ [ 255.486433] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing yml_node [ 261.210075] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 265.106473] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 267.754472] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 269.729469] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:13:27 EDT 2026 [ 286.357860] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 287.818138] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 305.120973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 305.127143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.160636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.194561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.196696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.232342] Lustre: Skipped 1 previous similar message [ 308.253021] Lustre: Skipped 3 previous similar messages [ 310.487375] Lustre: server umount lustre-MDT0000 complete [ 314.409990] LustreError: 6519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441254 with bad export cookie 259196014657222831 [ 314.432541] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.432798] LustreError: 6536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.432878] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.433265] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.447842] LustreError: 6536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 318.455824] Lustre: Skipped 2 previous similar messages [ 319.456053] LustreError: 6535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.481469] LustreError: 6535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 320.846825] Lustre: server umount lustre-MDT0001 complete [ 324.628724] Lustre: server umount lustre-OST0000 complete [ 328.594598] Lustre: server umount lustre-OST0001 complete [ 335.395238] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 344.732182] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 377.524195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.815199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 377.876150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.961580] Lustre: lustre-MDT0000: new disk, initializing [ 378.025233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.046971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 382.465159] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 392.460574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.605897] Lustre: 15093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 392.655537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 392.662263] Lustre: Skipped 1 previous similar message [ 392.727701] Lustre: lustre-MDT0001: new disk, initializing [ 392.839648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 392.891402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 392.899629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 397.110808] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 406.299734] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 412.009064] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 418.933580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.211330] Lustre: lustre-OST0000: new disk, initializing [ 419.216829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 419.234051] Lustre: 17060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 419.336670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.115665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 421.132550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 421.261978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.703932] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 435.154548] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 439.376720] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 439.577975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.601712] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.784091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.954110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.962070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.971683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.978304] Lustre: Skipped 4 previous similar messages [ 451.818585] Lustre: server umount lustre-OST0000 complete [ 457.189404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.199759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 457.202895] Lustre: Skipped 2 previous similar messages [ 457.220276] Lustre: Skipped 1 previous similar message [ 461.956933] Lustre: server umount lustre-MDT0000 complete [ 462.307787] LustreError: 15105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.332727] LustreError: 15105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 465.329171] LustreError: 18600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441405 with bad export cookie 259196014657224350 [ 465.333954] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.343269] LustreError: 18600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 465.819452] Lustre: server umount lustre-MDT0001 complete [ 469.465463] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:16:47 (1788441407) [ 471.647435] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 473.101267] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:16:51 (1788441411) [ 479.312359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.775070] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.845769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.006433] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 485.362872] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.560424] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.584189] LustreError: 19470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 494.798048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.698582] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 507.930955] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.368698] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 519.993247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 526.288348] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 532.924822] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 540.134452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.143074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.166642] LustreError: Skipped 1 previous similar message [ 540.167639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.190135] Lustre: Skipped 1 previous similar message [ 546.219778] Lustre: server umount lustre-OST0000 complete [ 556.416708] Lustre: server umount lustre-MDT0000 complete [ 560.290174] LustreError: 19450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441500 with bad export cookie 259196014657225232 [ 560.295448] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.325563] LustreError: 19450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 560.756649] Lustre: server umount lustre-MDT0001 complete [ 573.118800] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 576.408511] Key type lgssc unregistered [ 576.716792] LNet: 23001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.733886] LNetError: 23001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.754703] LNet: Removed LNI 192.168.203.135@tcp [ 577.620149] Key type .llcrypt unregistered [ 577.625673] Key type ._llcrypt unregistered [ 600.899417] Key type ._llcrypt registered [ 600.909430] Key type .llcrypt registered [ 601.035741] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 602.530834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.573117] alg: No test for adler32 (adler32-zlib) [ 603.711590] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 603.996594] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 605.696541] Key type lgssc registered [ 606.984098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.121134] loop: module loaded [ 615.193765] loop0: detected capacity change from 0 to 204800000 [ 615.387955] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.400851] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.823162] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.871084] loop0: detected capacity change from 0 to 204800000 [ 622.048422] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.074957] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 622.099482] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.663502] loop0: detected capacity change from 0 to 204800000 [ 632.268541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.286082] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.543924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 633.625840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 633.726047] Lustre: lustre-MDT0000: new disk, initializing [ 633.818222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.846971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 637.973488] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 646.022839] loop1: detected capacity change from 0 to 204800000 [ 651.484826] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 651.746110] Lustre: lustre-OST07c6: new disk, initializing [ 651.749907] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 651.753867] Lustre: Skipped 1 previous similar message [ 651.758629] Lustre: 26197:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 651.843900] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 657.111349] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 659.002088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 659.007866] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 659.043025] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 661.087461] Lustre: 26517:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 661.100922] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 661.107306] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 661.113789] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 661.121025] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 661.128628] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 678.453354] Lustre: 26517:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 678.462705] Lustre: 26517:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.473661] Lustre: 26517:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.482501] Lustre: 26517:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 678.491953] Lustre: 26517:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 678.501196] Lustre: 26517:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.176646] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 683.184282] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 683.189477] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 683.192561] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.195846] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.201018] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.205925] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 683.212306] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.218125] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 683.221476] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.234169] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.240335] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.415691] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 686.429154] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.441072] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 686.456060] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.472961] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.486742] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.496691] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 686.509489] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.525456] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.539551] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.545673] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.551399] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.537092] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 691.544198] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 691.552542] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 691.567941] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.577880] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 691.588152] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.602662] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 691.616508] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.627225] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 691.636031] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 691.639964] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 691.643232] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.855872] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 702.863950] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 702.867673] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 702.903146] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.923051] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.942894] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 702.979708] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 702.994768] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.006513] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 703.014187] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.030132] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.043735] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 754.891082] Lustre: server umount lustre-OST07c6 complete [ 773.203460] Lustre: server umount lustre-MDT0000 complete [ 793.149223] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 796.356921] Key type lgssc unregistered [ 796.659684] LNet: 28759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.668645] LNetError: 28759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.703928] LNet: Removed LNI 192.168.203.135@tcp [ 797.813170] Key type .llcrypt unregistered [ 797.816029] Key type ._llcrypt unregistered [ 816.811635] Key type ._llcrypt registered [ 816.814878] Key type .llcrypt registered [ 816.922839] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 818.039903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 818.098265] alg: No test for adler32 (adler32-zlib) [ 819.152621] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 819.421449] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 821.119269] Key type lgssc registered [ 822.374857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 830.849701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 830.875996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 832.303430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.557334] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 842.724384] LustreError: 30049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.735837] LustreError: 30049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 845.594766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.135627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 850.957582] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 858.572442] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 865.107531] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 871.679967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 872.313651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.500265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 880.701107] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 890.249197] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 906.210432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.216484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 906.233107] Lustre: Skipped 1 previous similar message [ 911.331121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 911.334119] Lustre: Skipped 1 previous similar message [ 911.857678] Lustre: server umount lustre-OST0000 complete [ 917.986937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 917.999489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.030894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.034139] Lustre: Skipped 1 previous similar message [ 921.571074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.571791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.292840] Lustre: server umount lustre-MDT0000 complete [ 926.693906] LustreError: 30049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.726859] LustreError: 30049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 929.027534] LustreError: 30031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441868 with bad export cookie 16164649862155021746 [ 929.061652] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.316242] Lustre: server umount lustre-MDT0001 complete [ 939.265074] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 941.977737] Key type lgssc unregistered [ 942.286894] LNet: 33790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.303181] LNetError: 33790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.340934] LNet: Removed LNI 192.168.203.135@tcp [ 944.365208] Key type .llcrypt unregistered [ 944.367110] Key type ._llcrypt unregistered [ 965.867941] Key type ._llcrypt registered [ 965.870620] Key type .llcrypt registered [ 965.971640] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 966.918701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.013143] alg: No test for adler32 (adler32-zlib) [ 968.161969] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 968.390823] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 970.015156] Key type lgssc registered [ 971.632750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.600600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.643574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.204544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.256272] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 990.690254] LustreError: 35063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.698609] LustreError: 35063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 992.472929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.872198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 997.499570] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1004.584923] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.592622] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1017.424480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1017.846128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.381623] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1026.037182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1032.264586] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1047.521771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.525197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.540589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1047.551194] Lustre: Skipped 1 previous similar message [ 1052.648192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.659764] Lustre: Skipped 1 previous similar message [ 1053.076414] Lustre: server umount lustre-OST0000 complete [ 1057.762833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.779064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.785224] Lustre: Skipped 2 previous similar messages [ 1057.799398] Lustre: Skipped 1 previous similar message [ 1062.889911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.902185] Lustre: Skipped 1 previous similar message [ 1063.382055] Lustre: server umount lustre-MDT0000 complete [ 1067.378407] LustreError: 35797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442007 with bad export cookie 16703879461163520417 [ 1067.385238] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.390909] LustreError: 35797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1067.746560] Lustre: server umount lustre-MDT0001 complete [ 1078.782266] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1081.674888] Key type lgssc unregistered [ 1082.029740] LNet: 38620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.038417] LNetError: 38620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.045411] LNet: Removed LNI 192.168.203.135@tcp [ 1083.976102] Key type .llcrypt unregistered [ 1083.979171] Key type ._llcrypt unregistered [ 1109.289527] Key type ._llcrypt registered [ 1109.293884] Key type .llcrypt registered [ 1109.440820] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1110.497937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.572693] alg: No test for adler32 (adler32-zlib) [ 1111.654484] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1112.049450] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1113.815165] Key type lgssc registered [ 1115.021325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1123.661267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1123.677223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.190519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.473463] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1135.588245] LustreError: 40040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.626525] LustreError: 40040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1139.577478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.093865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1145.068570] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1152.923942] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1158.370459] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1165.110625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.476601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.628182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1171.300582] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1178.431088] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1190.370090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.376598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.394894] Lustre: Skipped 1 previous similar message [ 1193.700581] Lustre: server umount lustre-MDT0000 complete [ 1195.491531] LustreError: 40035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1195.509428] LustreError: 40035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1197.605166] LustreError: 40020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442137 with bad export cookie 12473335875430808716 [ 1197.613632] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.619188] LustreError: 40020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1200.608333] LustreError: 41432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.612366] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.626210] LustreError: 41432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1200.650895] Lustre: Skipped 1 previous similar message [ 1200.659884] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1200.668296] Lustre: Skipped 3 previous similar messages [ 1202.657571] LustreError: 40035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.658691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1202.689618] LustreError: 40035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1202.715109] Lustre: Skipped 1 previous similar message [ 1203.994792] Lustre: server umount lustre-MDT0001 complete [ 1224.986865] Lustre: server umount lustre-OST0000 complete [ 1241.953519] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1245.073706] Key type lgssc unregistered [ 1245.414847] LNet: 43903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.423268] LNetError: 43903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.440391] LNet: Removed LNI 192.168.203.135@tcp [ 1246.305381] Key type .llcrypt unregistered [ 1246.315116] Key type ._llcrypt unregistered [ 1271.632540] Key type ._llcrypt registered [ 1271.639238] Key type .llcrypt registered [ 1271.809355] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1273.183607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.297581] alg: No test for adler32 (adler32-zlib) [ 1274.565202] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1274.899422] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1276.631118] Key type lgssc registered [ 1278.028447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1285.874385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1285.900793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.442433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.925977] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1297.890857] LustreError: 45327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.926181] LustreError: 45327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1301.231340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.632434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1306.868188] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1314.928800] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1320.796351] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1328.219468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1328.529076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.669303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1335.168020] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1343.029844] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1352.170531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.172638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1352.190041] Lustre: Skipped 1 previous similar message [ 1352.207286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.110305] Lustre: server umount lustre-OST0000 complete [ 1372.650766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.663572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1372.672956] Lustre: Skipped 2 previous similar messages [ 1373.728825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.737357] Lustre: Skipped 1 previous similar message [ 1378.372732] Lustre: server umount lustre-MDT0000 complete [ 1378.788923] LustreError: 46056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.816589] LustreError: 46056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1382.182990] LustreError: 48286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442322 with bad export cookie 14775209821316794040 [ 1382.192024] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1382.208375] LustreError: 48286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1388.843630] Lustre: server umount lustre-MDT0001 complete [ 1402.364291] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1404.747785] Key type lgssc unregistered [ 1404.951549] LNet: 49043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.959692] LNetError: 49043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.987290] LNet: Removed LNI 192.168.203.135@tcp [ 1405.697913] Key type .llcrypt unregistered [ 1405.702279] Key type ._llcrypt unregistered [ 1428.327262] Key type ._llcrypt registered [ 1428.331343] Key type .llcrypt registered [ 1428.471794] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1429.459573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1429.604211] alg: No test for adler32 (adler32-zlib) [ 1430.670198] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1430.894771] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1432.567417] Key type lgssc registered [ 1433.760030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.513946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1441.531345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.992644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.249972] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1452.517296] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.529531] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1455.846900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.250615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.123789] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1468.746962] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1474.456481] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1481.407348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.811496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.003631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1488.353589] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1496.119156] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1505.610526] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1508.903871] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1521.675434] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h34m18s [ 1523.303279] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1527.265120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.272765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.293684] Lustre: Skipped 1 previous similar message [ 1527.313814] Lustre: Skipped 2 previous similar messages [ 1528.359857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1529.482632] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 1529.490152] Lustre: Skipped 2 previous similar messages [ 1531.679279] Lustre: server umount lustre-MDT0000 complete [ 1534.438218] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.496129] LustreError: 50457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1535.908334] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1536.628062] LustreError: 50458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.647255] LustreError: 53222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1540.667456] LustreError: 53222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1542.927647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.033675] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.373799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.470023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1544.369505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1548.873721] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1552.485976] Lustre: DEBUG MARKER: Wait for df (36757) ... [ 1554.043673] Lustre: DEBUG MARKER: done [ 1557.985324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1557.991991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.006829] Lustre: Skipped 1 previous similar message [ 1558.012729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1558.018284] Lustre: Skipped 1 previous similar message [ 1559.010258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1559.020537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.157929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1565.165720] Lustre: Skipped 4 previous similar messages [ 1572.322369] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1572.495737] Lustre: server umount lustre-OST0000 complete [ 1580.521121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1580.525570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.536202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.562174] Lustre: Skipped 1 previous similar message [ 1580.567039] Lustre: Skipped 2 previous similar messages [ 1582.500841] Lustre: server umount lustre-MDT0000 complete [ 1586.638346] LustreError: 50441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442526 with bad export cookie 18310571080217023672 [ 1586.646072] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.660644] LustreError: 50441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1587.277278] Lustre: server umount lustre-MDT0001 complete [ 1600.632342] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1603.712830] Key type lgssc unregistered [ 1604.119891] LNet: 55390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.133360] LNetError: 55390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1604.149691] LNet: Removed LNI 192.168.203.135@tcp [ 1605.208051] Key type .llcrypt unregistered [ 1605.215322] Key type ._llcrypt unregistered [ 1644.105078] Key type ._llcrypt registered [ 1644.110660] Key type .llcrypt registered [ 1644.227051] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1645.323504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.353572] alg: No test for adler32 (adler32-zlib) [ 1646.402691] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1646.620094] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1648.263173] Key type lgssc registered [ 1649.446736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1658.573505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1658.594539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.929843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1659.962230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1660.105460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.521406] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1674.034959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.265137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1674.323549] Lustre: 57596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1674.790140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1679.915493] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1688.360560] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1694.071709] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1701.003799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1701.322526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1701.541958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.921894] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1709.128248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1713.829792] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1720.859046] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h37m39s [ 1722.474798] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1729.046891] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1731.704030] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1731.806355] LustreError: 59930:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1752.252860] Lustre: lustre-MDT0000: Client ac71a690-05cf-4342-81f2-300dc6b4aecf (at 192.168.203.35@tcp) reconnecting [ 1771.847121] LustreError: 59930:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1773.893688] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1776.140353] Lustre: DEBUG MARKER: done [ 1784.018321] Lustre: server umount lustre-OST0000 complete [ 1790.946257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.972941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.983384] Lustre: Skipped 1 previous similar message [ 1792.047813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.056996] Lustre: Skipped 1 previous similar message [ 1798.119920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.125823] Lustre: Skipped 1 previous similar message [ 1801.183371] Lustre: 56921:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442725/real 1788442725] req@0000000094f2d930 x1875317973114496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788442741 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1801.211890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.224303] Lustre: Skipped 1 previous similar message [ 1802.207133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1802.275469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.284830] Lustre: Skipped 1 previous similar message [ 1802.521751] Lustre: server umount lustre-MDT0000 complete [ 1806.035697] LustreError: 57589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442745 with bad export cookie 1214950287124711939 [ 1806.043193] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.044060] LustreError: 57589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1806.351349] Lustre: server umount lustre-MDT0001 complete [ 1816.016090] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 1818.824282] Key type lgssc unregistered [ 1819.151311] LNet: 61933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.173976] LNetError: 61933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.225373] LNet: Removed LNI 192.168.203.135@tcp [ 1820.193265] Key type .llcrypt unregistered [ 1820.194850] Key type ._llcrypt unregistered [ 1860.982215] Key type ._llcrypt registered [ 1860.984871] Key type .llcrypt registered [ 1861.111944] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 1861.914342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.144211] alg: No test for adler32 (adler32-zlib) [ 1863.359699] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1863.704795] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 1865.439143] Key type lgssc registered [ 1866.665857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1870.097581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1870.128876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.423595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1871.464950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1871.593345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.656681] Lustre: Failing over lustre-MDT0000 [ 1872.978382] Lustre: server umount lustre-MDT0000 complete [ 1879.197830] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:40:17 (1788442817) [ 1885.959676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.417652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.688634] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1899.296037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.440396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1899.501459] Lustre: 64731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1899.759780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1904.467358] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1912.308675] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1917.568582] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.049926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1924.281916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1924.486350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.654657] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1932.536098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1940.360200] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.484152] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1953.255751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.260884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.279412] Lustre: Skipped 2 previous similar messages [ 1953.286613] Lustre: Skipped 1 previous similar message [ 1958.370108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.375622] Lustre: Skipped 3 previous similar messages [ 1959.196990] Lustre: server umount lustre-MDT0000 complete [ 1962.497488] LustreError: 65468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442902 with bad export cookie 14764629984301965528 [ 1962.501516] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1962.506382] LustreError: 65468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1962.712606] Lustre: server umount lustre-MDT0001 complete [ 1965.405592] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1969.686835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1979.875560] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442903/real 1788442903] req@00000000a6939f42 x1875318200638336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1979.911943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.773709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.032094] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442907/real 1788442907] req@000000002c01afdd x1875318200638464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442923 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1987.854127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.081790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1988.777915] Lustre: 63658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442912/real 1788442912] req@00000000d2190f5e x1875318200638848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.815853] Lustre: 63658:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1993.045941] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 1994.015143] Lustre: 63656:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442917/real 1788442917] req@000000002ba50d74 x1875318200638976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1996.266877] LustreError: 68504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.280907] LustreError: 68504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2001.388787] LustreError: 68503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.424107] LustreError: 68503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2001.578641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.976210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2005.667521] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2007.020176] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2009.061929] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2012.311606] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2017.773187] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2025.135545] Lustre: server umount lustre-MDT0000 complete [ 2027.489088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2027.492992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.502151] LustreError: 69558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.988411] LustreError: 65468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442968 with bad export cookie 14764629984301967803 [ 2028.991449] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.994697] LustreError: 65468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2029.421225] Lustre: server umount lustre-MDT0001 complete [ 2036.779300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2045.413089] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442969/real 1788442969] req@0000000016b55c50 x1875318200664704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2045.439978] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.025620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.091439] LustreError: 66583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.219787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.299241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2055.263952] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442978/real 1788442978] req@00000000f784567a x1875318200665344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.278060] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2055.299968] Lustre: 63655:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2059.149648] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2060.776041] LustreError: 66583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.802497] LustreError: 66583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2068.158355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.596456] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2073.849754] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2082.326919] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2088.729686] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2095.073506] Lustre: server umount lustre-MDT0000 complete [ 2098.901887] LustreError: 64726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443038 with bad export cookie 14764629984301969182 [ 2098.922675] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.175716] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.193876] Lustre: Skipped 1 previous similar message [ 2099.215199] LustreError: 71184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.251088] LustreError: 71184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2099.372757] Lustre: server umount lustre-MDT0001 complete [ 2106.341678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2109.915921] Lustre: DEBUG MARKER: files compared the same [ 2112.331933] Lustre: server umount lustre-OST0000 complete [ 2129.437310] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2132.123794] Key type lgssc unregistered [ 2132.527664] LNet: 74463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.535219] LNetError: 74463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.553686] LNet: Removed LNI 192.168.203.135@tcp [ 2133.478407] Key type .llcrypt unregistered [ 2133.481313] Key type ._llcrypt unregistered [ 2159.218643] Key type ._llcrypt registered [ 2159.223239] Key type .llcrypt registered [ 2159.339380] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2160.486239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.631032] alg: No test for adler32 (adler32-zlib) [ 2161.747273] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2162.076861] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2163.767211] Key type lgssc registered [ 2165.003870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2172.941811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2172.953957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.457829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2179.014993] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug +malloc all [ 2184.674234] LustreError: 75882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.684041] LustreError: 75882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2187.277366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.551380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2192.081162] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug +malloc all [ 2199.882235] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2205.509880] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2212.264417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2212.639145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.795767] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug +malloc all [ 2219.841837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2227.590362] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2243.553710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2243.555561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.590817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.007128] Lustre: server umount lustre-OST0000 complete [ 2253.791889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.808670] Lustre: Skipped 1 previous similar message [ 2253.813270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.815600] Lustre: Skipped 2 previous similar messages [ 2257.785835] Lustre: server umount lustre-MDT0000 complete [ 2258.913249] LustreError: 76610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.943480] LustreError: 76610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2261.518736] LustreError: 75863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443201 with bad export cookie 12547872450774067109 [ 2261.530334] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.531409] LustreError: 75863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2262.055949] Lustre: server umount lustre-MDT0001 complete [ 2273.467707] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2276.560621] Key type lgssc unregistered [ 2276.889269] LNet: 79437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.906938] LNetError: 79437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.929771] LNet: Removed LNI 192.168.203.135@tcp [ 2277.974995] Key type .llcrypt unregistered [ 2277.978193] Key type ._llcrypt unregistered [ 2300.854729] Key type ._llcrypt registered [ 2300.857719] Key type .llcrypt registered [ 2300.956505] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2302.208596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.268651] alg: No test for adler32 (adler32-zlib) [ 2303.367726] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2303.601188] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2305.304781] Key type lgssc registered [ 2306.377629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2313.909936] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2313.936209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.697983] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2334.815320] LustreError: 80830:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.135@tcp: failed processing log, type 4: rc = -110 [ 2361.503216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.909857] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2376.005354] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2384.580818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.735595] Lustre: *** cfs_fail_loc=706, val=0*** [ 2385.052845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.144571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2389.656689] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2390.498144] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.520423] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2395.622828] LustreError: 80856:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.645953] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.679074] LustreError: 80855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2398.206406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.749136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.636780] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2409.337816] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2414.678825] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2419.170140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.173035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.196166] Lustre: Skipped 1 previous similar message [ 2419.224335] Lustre: Skipped 1 previous similar message [ 2422.970804] Lustre: server umount lustre-OST0000 complete [ 2429.411612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.428992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2429.441791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.458237] Lustre: Skipped 1 previous similar message [ 2433.075595] Lustre: server umount lustre-MDT0000 complete [ 2434.529145] LustreError: 82563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.538367] LustreError: 82563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2436.138399] LustreError: 80838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443376 with bad export cookie 962097848618734709 [ 2436.142979] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.150088] LustreError: 80838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2436.424060] Lustre: server umount lustre-MDT0001 complete [ 2446.168521] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2448.931084] Key type lgssc unregistered [ 2449.197687] LNet: 84448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2449.201438] LNetError: 84448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.224753] LNet: Removed LNI 192.168.203.135@tcp [ 2450.188330] Key type .llcrypt unregistered [ 2450.191243] Key type ._llcrypt unregistered [ 2473.843528] Key type ._llcrypt registered [ 2473.846243] Key type .llcrypt registered [ 2473.943912] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2475.023631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.112465] alg: No test for adler32 (adler32-zlib) [ 2476.139802] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2476.350650] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2478.047193] Key type lgssc registered [ 2479.292162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.078583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.099325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.550382] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2498.250040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.763566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2503.359424] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2504.161473] LustreError: 86552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.191059] LustreError: 86552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2508.260741] LustreError: 86550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.275528] LustreError: 86550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2511.105108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2511.370107] LustreError: 86552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.392932] LustreError: 86552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2511.394555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.388539] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2516.452869] LustreError: 86551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.465026] LustreError: 86551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2521.569670] LustreError: 86555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.591586] LustreError: 86555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2523.043441] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.055437] Lustre: MGS: Client 68643213-d865-4a00-929e-2d598587e35e (at 0@lo) reconnecting [ 2523.073550] LustreError: 86566:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2523.272456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.345449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2526.861818] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2540.002088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.016390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.020963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2543.010148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.020595] Lustre: Skipped 2 previous similar messages [ 2545.725976] Lustre: server umount lustre-OST0000 complete [ 2550.241222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2550.245949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.253511] Lustre: Skipped 1 previous similar message [ 2550.257360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.312672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.313943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.341060] Lustre: Skipped 1 previous similar message [ 2555.902755] Lustre: server umount lustre-MDT0000 complete [ 2558.435102] LustreError: 88559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.551646] Lustre: server umount lustre-MDT0001 complete [ 2562.689887] Lustre: server umount MGS complete [ 2573.084570] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:51:51 (1788443511) [ 2601.537301] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 2616.184864] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2654.400824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.434776] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2667.954106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.161363] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2668.192685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2668.244636] Lustre: lustre-MDT0001: new disk, initializing [ 2668.327571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2668.355316] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2668.358332] Lustre: 93736:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2668.379533] LustreError: 93736:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2671.311201] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2680.893644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2681.164723] Lustre: lustre-OST0000: new disk, initializing [ 2681.173603] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2681.191576] Lustre: 94504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2681.269266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.197561] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2689.135994] Lustre: 94843:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d296bc0d x1875318843415808/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788443645 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2698.628587] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2705.232473] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.244176] Lustre: MGS: Client 8422ae9b-556a-4aa1-abb5-fc20adbf099e (at 0@lo) reconnecting [ 2705.293411] Lustre: 93004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2705.311677] LustreError: 93722:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2705.373136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2705.443979] Lustre: lustre-MDT0000: new disk, initializing [ 2705.582406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.606611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2705.622618] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2706.656918] Lustre: 95573:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008c78c3e6 x1875318843422592/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788443662 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2709.564995] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2711.904293] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2715.760390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2715.792702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2715.996238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2740.705965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.717836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.728174] Lustre: Skipped 1 previous similar message [ 2740.743840] Lustre: Skipped 1 previous similar message [ 2746.623160] Lustre: server umount lustre-OST0000 complete [ 2756.066221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2756.070888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.087081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2756.095353] Lustre: Skipped 1 previous similar message [ 2756.122598] Lustre: Skipped 3 previous similar messages [ 2757.413257] Lustre: server umount lustre-MDT0000 complete [ 2761.184415] LustreError: 93712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.192034] LustreError: 93712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2768.203928] Lustre: server umount lustre-MDT0001 complete [ 2773.391579] Lustre: server umount MGS complete [ 2787.286585] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:55:25 (1788443725) [ 2806.111543] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2808.611269] Key type lgssc unregistered [ 2808.919496] LNet: 98073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2808.925415] LNetError: 98073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2808.941481] LNet: Removed LNI 192.168.203.135@tcp [ 2810.074041] Key type .llcrypt unregistered [ 2810.078452] Key type ._llcrypt unregistered [ 2832.791810] Key type ._llcrypt registered [ 2832.796662] Key type .llcrypt registered [ 2832.928668] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2834.144497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2834.208266] alg: No test for adler32 (adler32-zlib) [ 2835.300283] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2835.782967] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2837.680053] Key type lgssc registered [ 2839.068856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2846.985730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2847.016510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.688281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.610909] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2858.977788] LustreError: 99296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.985622] LustreError: 99296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2860.390742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.960172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2865.598216] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2873.115818] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2878.159431] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2884.802215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.259822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.361425] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2893.504735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2902.262594] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2912.230290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.252904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2912.281241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.275725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.284757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2914.303466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.867383] Lustre: server umount lustre-OST0000 complete [ 2922.465663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2922.473850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.484386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.487090] Lustre: Skipped 1 previous similar message [ 2924.512161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2924.516555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.859507] Lustre: server umount lustre-MDT0000 complete [ 2929.637203] LustreError: 99297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.654679] LustreError: 99297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2930.891733] LustreError: 99277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443870 with bad export cookie 13732737080472520942 [ 2930.894393] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2930.900878] LustreError: 99277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2931.293631] Lustre: server umount lustre-MDT0001 complete [ 2942.534929] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 2946.076210] Key type lgssc unregistered [ 2946.391882] LNet: 102833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2946.402856] LNetError: 102833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2946.454774] LNet: Removed LNI 192.168.203.135@tcp [ 2947.348217] Key type .llcrypt unregistered [ 2947.350459] Key type ._llcrypt unregistered [ 2964.726591] Key type ._llcrypt registered [ 2964.731649] Key type .llcrypt registered [ 2964.869029] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 2966.257060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2966.378514] alg: No test for adler32 (adler32-zlib) [ 2967.418477] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2967.688495] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 2969.440856] Key type lgssc registered [ 2970.625901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2977.113433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2977.120875] LustreError: 103917:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2977.126903] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2977.133797] LustreError: 103917:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2977.144362] LustreError: 103917:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2977.149934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.154362] LustreError: 103917:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2978.765337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.535875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.874411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2991.025545] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 2996.636939] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing lsmod [ 2999.822899] LustreError: 105112:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2999.866728] LustreError: 105123:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2999.884308] LustreError: 105112:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 2999.897695] LustreError: 105112:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2999.924443] LustreError: 105112:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2999.939874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3000.453283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.601100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3007.473651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.480441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.488989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3008.994915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.000480] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.024855] Lustre: Skipped 1 previous similar message [ 3011.112497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.132768] Lustre: Skipped 2 previous similar messages [ 3016.169906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.188531] Lustre: Skipped 2 previous similar messages [ 3018.207675] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3018.469204] Lustre: server umount lustre-MDT0000 complete [ 3021.284460] LustreError: 105309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.312646] LustreError: 105309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3022.346824] LustreError: 103934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443962 with bad export cookie 8754394442180718413 [ 3022.349333] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.357643] LustreError: 103934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3023.455257] LustreError: 104387:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000eb6efd7f x1875319358249472/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3023.598776] Lustre: server umount lustre-MDT0001 complete [ 3038.969363] Lustre: server umount lustre-OST0000 complete [ 3047.602779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.168601] LustreError: 106189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.322847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.377868] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3053.539570] LustreError: 106190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.660383] LustreError: 106189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.601309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.965631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.409080] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3077.359988] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.470152] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3098.820231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.812981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.787620] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3116.116284] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3118.261335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3133.920797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.925475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.926518] LustreError: Skipped 1 previous similar message [ 3133.927202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.962234] Lustre: Skipped 1 previous similar message [ 3135.496821] Lustre: server umount lustre-OST0000 complete [ 3140.066279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3140.079630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.165385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.170746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.178785] Lustre: Skipped 2 previous similar messages [ 3145.714347] Lustre: server umount lustre-MDT0000 complete [ 3149.302183] LustreError: 106190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.338775] LustreError: 106190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3150.433127] LustreError: 106170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444090 with bad export cookie 8754394442180719330 [ 3150.443414] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3150.916722] Lustre: server umount lustre-MDT0001 complete [ 3164.786337] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 3168.853053] Key type lgssc unregistered [ 3169.325729] LNet: 109765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.334962] LNetError: 109765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3169.376966] LNet: Removed LNI 192.168.203.135@tcp [ 3170.497134] Key type .llcrypt unregistered [ 3170.500335] Key type ._llcrypt unregistered [ 3201.648810] Key type ._llcrypt registered [ 3201.651311] Key type .llcrypt registered [ 3201.767517] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3203.052819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3203.086312] alg: No test for adler32 (adler32-zlib) [ 3204.131512] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3204.372185] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 3206.047247] Key type lgssc registered [ 3207.430852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3215.028846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3215.067358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.832241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.346485] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3227.137678] LustreError: 111182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.153336] LustreError: 111182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3231.697953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.057300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3236.815925] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3247.005701] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3254.062565] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3262.151393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3262.589189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.614699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3268.810293] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3276.447273] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3282.351139] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3285.706985] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3292.270676] Lustre: server umount lustre-OST0000 complete [ 3297.765578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.773803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3297.798257] Lustre: Skipped 1 previous similar message [ 3297.823175] Lustre: Skipped 1 previous similar message [ 3302.880913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.896756] Lustre: Skipped 1 previous similar message [ 3306.058550] Lustre: server umount lustre-MDT0000 complete [ 3308.003941] LustreError: 111181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.044394] LustreError: 111181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3309.028468] Lustre: 110492:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444232/real 1788444232] req@0000000090fcf6ea x1875319606775552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444248 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3309.067822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.089960] LustreError: 111177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.117895] LustreError: 111177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3310.054499] LustreError: 111161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444249 with bad export cookie 1518740981089518164 [ 3310.055775] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.082641] LustreError: 111161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3310.462440] Lustre: server umount lustre-MDT0001 complete [ 3320.880914] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 3323.769677] Key type lgssc unregistered [ 3324.025593] LNet: 114821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3324.042492] LNetError: 114821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3324.073686] LNet: Removed LNI 192.168.203.135@tcp [ 3324.974273] Key type .llcrypt unregistered [ 3324.976479] Key type ._llcrypt unregistered [ 3341.549576] Key type ._llcrypt registered [ 3341.554343] Key type .llcrypt registered [ 3341.669697] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3343.194326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.256460] alg: No test for adler32 (adler32-zlib) [ 3344.427494] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3344.787678] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 3346.527215] Key type lgssc registered [ 3347.690758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3356.029668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3356.070556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.750390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.793746] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3367.910155] LustreError: 116041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.935663] LustreError: 116041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3372.880341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.043590] LustreError: 116042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.287324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.886509] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3386.313438] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3392.122453] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3398.804627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.079821] LustreError: 117865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.432594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.552362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3405.519652] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3413.374537] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3423.200656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.221544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.226870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.275565] Lustre: Skipped 1 previous similar message [ 3428.322212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.338984] Lustre: Skipped 2 previous similar messages [ 3429.005635] Lustre: server umount lustre-OST0000 complete [ 3434.993852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3435.004814] LustreError: Skipped 1 previous similar message [ 3435.011146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.032247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.568042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.570980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3440.141617] Lustre: server umount lustre-MDT0000 complete [ 3443.689929] LustreError: 116035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.728832] LustreError: 116035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3444.431860] LustreError: 116023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444384 with bad export cookie 9406646211903174740 [ 3444.439824] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3444.442974] LustreError: 116023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3444.970788] Lustre: server umount lustre-MDT0001 complete [ 3458.919266] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 3461.904923] Key type lgssc unregistered [ 3462.185693] LNet: 119582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3462.194385] LNetError: 119582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3462.224632] LNet: Removed LNI 192.168.203.135@tcp [ 3463.094821] Key type .llcrypt unregistered [ 3463.097478] Key type ._llcrypt unregistered [ 3488.565366] Key type ._llcrypt registered [ 3488.569236] Key type .llcrypt registered [ 3488.724904] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3489.915309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3489.987383] alg: No test for adler32 (adler32-zlib) [ 3491.132743] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3491.353435] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 3493.015326] Key type lgssc registered [ 3494.103586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3502.835142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3502.852385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.411630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3509.531084] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3514.854716] LustreError: 121000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.894629] LustreError: 121000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3518.953960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.411544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3524.555134] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3532.899901] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3538.650764] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3545.535957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3546.002637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.130817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3552.567581] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3562.524537] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3572.758230] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3576.902978] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3576.905890] Lustre: Skipped 2 previous similar messages [ 3579.078725] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3579.126494] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3581.852519] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3581.862429] Lustre: Skipped 2 previous similar messages [ 3587.711343] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3587.716862] Lustre: Skipped 1 previous similar message [ 3593.504951] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.35@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3593.518913] Lustre: Skipped 2 previous similar messages [ 3593.624481] Lustre: lustre-MDT0000: nosquash_nids set to 2@gni 0@lo 192.168.203.35@tcp 192.168.0.[2,10]@tcp [ 3599.237024] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.35/24@tcp in log lustre-MDT0000 [ 3599.243878] Lustre: Skipped 2 previous similar messages [ 3600.418882] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.35/24@tcp [ 3610.594839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.599669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3610.626242] Lustre: Skipped 1 previous similar message [ 3610.642286] Lustre: Skipped 1 previous similar message [ 3615.720459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3615.735752] Lustre: Skipped 1 previous similar message [ 3616.489170] Lustre: server umount lustre-OST0000 complete [ 3620.833553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.835073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.848609] Lustre: Skipped 1 previous similar message [ 3620.866229] Lustre: Skipped 1 previous similar message [ 3625.954913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.976693] Lustre: Skipped 1 previous similar message [ 3626.543576] Lustre: server umount lustre-MDT0000 complete [ 3630.218233] LustreError: 124038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444570 with bad export cookie 942785686472149366 [ 3630.234869] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.251879] LustreError: 124038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3630.553351] Lustre: server umount lustre-MDT0001 complete [ 3642.150529] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 3646.092683] Key type lgssc unregistered [ 3646.497830] LNet: 125750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3646.516685] LNetError: 125750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3646.553711] LNet: Removed LNI 192.168.203.135@tcp [ 3648.112525] Key type .llcrypt unregistered [ 3648.115608] Key type ._llcrypt unregistered [ 3678.377323] Key type ._llcrypt registered [ 3678.380415] Key type .llcrypt registered [ 3678.542376] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3679.840144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3679.946699] alg: No test for adler32 (adler32-zlib) [ 3681.158857] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3681.487101] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 3683.167326] Key type lgssc registered [ 3684.397590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3692.656952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3692.677738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.283023] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3694.287411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.318403] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.35/24@tcp [ 3698.954701] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3704.802744] LustreError: 127312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.818448] LustreError: 127312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3708.187460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.667818] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3708.713539] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.35/24@tcp [ 3708.775362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3713.737758] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3721.835350] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3727.719851] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3735.185807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.556314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3739.718440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3741.742752] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3749.359427] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3764.201483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.205844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.212098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.220037] Lustre: Skipped 1 previous similar message [ 3764.270380] Lustre: Skipped 1 previous similar message [ 3769.314803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.324407] Lustre: Skipped 1 previous similar message [ 3769.659851] Lustre: server umount lustre-OST0000 complete [ 3774.434155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3774.453469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.467158] Lustre: Skipped 1 previous similar message [ 3779.552661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.567413] Lustre: Skipped 1 previous similar message [ 3779.837429] Lustre: server umount lustre-MDT0000 complete [ 3783.555160] LustreError: 127292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444723 with bad export cookie 118504954102692390 [ 3783.561883] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.566732] LustreError: 127292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3784.024600] Lustre: server umount lustre-MDT0001 complete [ 3796.096604] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 3799.247769] Key type lgssc unregistered [ 3799.630600] LNet: 130904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3799.641540] LNetError: 130904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3799.686741] LNet: Removed LNI 192.168.203.135@tcp [ 3801.280771] Key type .llcrypt unregistered [ 3801.283127] Key type ._llcrypt unregistered [ 3830.719493] Key type ._llcrypt registered [ 3830.721810] Key type .llcrypt registered [ 3830.833297] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 3846.295460] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 3847.750229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3848.172607] alg: No test for adler32 (adler32-zlib) [ 3849.332843] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3849.611358] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 3851.303174] Key type lgssc registered [ 3852.583631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3889.854747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3889.882224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.122702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3891.173530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3891.284316] Lustre: lustre-MDT0000: new disk, initializing [ 3891.372761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.396304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3895.739661] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3906.043680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.171358] Lustre: 134305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3906.220281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3906.226274] Lustre: Skipped 1 previous similar message [ 3906.297737] Lustre: lustre-MDT0001: new disk, initializing [ 3906.383544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3906.411606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3906.427781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3911.224120] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3920.777077] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3926.543227] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3932.651638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3932.997336] Lustre: lustre-OST0000: new disk, initializing [ 3933.001176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3933.007965] Lustre: 136274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3933.129807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.935905] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 3940.425767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3940.435867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3940.588858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3949.750816] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3955.048662] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3955.334746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3960.175742] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3960.479883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3966.434633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.435601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.437131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.450912] Lustre: Skipped 1 previous similar message [ 3971.553530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.570027] Lustre: Skipped 2 previous similar messages [ 3976.685356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3976.698522] Lustre: Skipped 1 previous similar message [ 3977.183416] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3977.782950] Lustre: server umount lustre-OST0000 complete [ 3986.914387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.927323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.936949] Lustre: Skipped 1 previous similar message [ 3992.035616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3992.043269] Lustre: Skipped 1 previous similar message [ 3998.175232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3998.357172] Lustre: server umount lustre-MDT0000 complete [ 4002.840755] LustreError: 135100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444942 with bad export cookie 17605385746600262996 [ 4002.853157] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4002.854487] LustreError: 135100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4003.257535] Lustre: server umount lustre-MDT0001 complete [ 4013.774028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.187283] LustreError: 138492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.207364] LustreError: 138492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4014.259550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.000263] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4019.680263] LustreError: 138493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.807281] LustreError: 138492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4029.204729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.690341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4035.498091] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4044.328547] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4051.296668] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4058.009384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4058.399137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.484461] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4071.584117] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4076.826028] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4077.149381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4081.938231] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4082.062688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4090.436187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.669970] Lustre: lustre-OST0001: new disk, initializing [ 4090.674709] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4090.680627] Lustre: 141865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4090.807366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4098.681647] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4099.855995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4099.868633] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4100.073889] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4110.222764] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4110.606276] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4116.805334] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4117.208119] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4122.613350] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8888c8b19000.ost_server_uuid 50 [ 4124.806583] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8888c8b19000.ost_server_uuid in IDLE state after 0 sec [ 4135.924242] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4135.929335] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.931084] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4135.931090] Lustre: Skipped 2 previous similar messages [ 4135.931996] LustreError: Skipped 1 previous similar message [ 4135.980391] Lustre: Skipped 2 previous similar messages [ 4137.665104] Lustre: server umount lustre-OST0001 complete [ 4141.029834] LustreError: 142104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.040853] LustreError: 142104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4141.536267] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.543207] LustreError: Skipped 1 previous similar message [ 4141.543725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.564881] Lustre: Skipped 1 previous similar message [ 4146.155087] LustreError: 140316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.177122] LustreError: 140316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4147.950227] Lustre: server umount lustre-OST0000 complete [ 4152.802301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4152.813729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.836373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4152.844754] Lustre: Skipped 5 previous similar messages [ 4157.963744] Lustre: server umount lustre-MDT0000 complete [ 4161.505378] LustreError: 138492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.523780] LustreError: 138492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4161.539502] LustreError: 138474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445101 with bad export cookie 17605385746600263878 [ 4161.546159] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4161.553960] LustreError: 138474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4162.044962] Lustre: server umount lustre-MDT0001 complete [ 4185.491450] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 4188.985871] Key type lgssc unregistered [ 4189.304534] LNet: 144874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4189.315533] LNetError: 144874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4189.335660] LNet: Removed LNI 192.168.203.135@tcp [ 4190.166282] Key type .llcrypt unregistered [ 4190.178288] Key type ._llcrypt unregistered [ 4231.044797] Key type ._llcrypt registered [ 4231.047659] Key type .llcrypt registered [ 4231.155711] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 4248.616709] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 4250.052415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4250.119522] alg: No test for adler32 (adler32-zlib) [ 4251.123510] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4251.289100] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 4252.935379] Key type lgssc registered [ 4254.976095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4291.838483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4291.886298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.188597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4293.215878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4293.324164] Lustre: lustre-MDT0000: new disk, initializing [ 4293.459633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.484503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4298.140904] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4308.540826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.700975] Lustre: 149033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4308.772936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4308.779584] Lustre: Skipped 1 previous similar message [ 4308.838576] Lustre: lustre-MDT0001: new disk, initializing [ 4308.905369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4308.949416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4308.965604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4313.865109] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4322.967708] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4327.433446] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4333.959782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4334.180713] Lustre: lustre-OST0000: new disk, initializing [ 4334.183891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4334.189481] Lustre: 151004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4334.268527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.827593] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4342.966372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4343.026803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4343.331836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4354.152624] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4358.292991] Lustre: Failing over lustre-OST0000 [ 4358.403134] Lustre: server umount lustre-OST0000 complete [ 4375.007487] Lustre: 147259:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445298/real 1788445298] req@00000000fae56507 x1875320704632064/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445314 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4375.008715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.041628] Lustre: 147259:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4377.567148] Lustre: 147259:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445301/real 1788445301] req@00000000cbc47b83 x1875320704633088/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788445317 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4377.814063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4378.222668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4378.263584] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4379.235398] Lustre: 147258:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445303/real 1788445303] req@00000000fa0f5041 x1875320704634112/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4379.279136] Lustre: 147258:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4384.226093] Lustre: 147259:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445308/real 1788445308] req@000000001fd757b0 x1875320704635264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4384.359352] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4384.532524] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4386.463500] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4386.468942] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4388.103767] Lustre: Failing over lustre-MDT0000 [ 4388.312746] Lustre: server umount lustre-MDT0000 complete [ 4389.151323] Lustre: 147259:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445313/real 1788445313] req@000000003faa83c1 x1875320704636288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4389.163837] Lustre: 147259:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4389.862460] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.886788] Lustre: Skipped 3 previous similar messages [ 4394.976944] LustreError: 149043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.005618] LustreError: 149043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4396.656281] LustreError: 149042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.664205] LustreError: 149041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.704268] Lustre: 149039:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445328/real 1788445328] req@00000000cd710373 x1875320704645632/t0(0) o101->MGC192.168.203.135@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788445344 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4404.718639] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.726849] Lustre: 149039:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.135@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4404.734197] LustreError: 150406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.734207] LustreError: 150406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4407.900732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4409.825404] LustreError: 149042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.851158] LustreError: 149042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4414.945651] LustreError: 147257:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fbcacc7e x1875320704651264/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4415.282377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.351991] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4418.167838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4419.282825] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4420.598503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4420.615650] Lustre: Skipped 1 previous similar message [ 4420.660552] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4420.747916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4430.822398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.825100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4430.851816] Lustre: Skipped 1 previous similar message [ 4434.042556] Lustre: server umount lustre-OST0000 complete [ 4441.060087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4441.061404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4441.091768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4441.115853] Lustre: Skipped 2 previous similar messages [ 4444.783044] Lustre: server umount lustre-MDT0000 complete [ 4446.177870] LustreError: 149919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.194497] LustreError: 149919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4448.352308] LustreError: 149028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445388 with bad export cookie 16628616400211975837 [ 4448.358330] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4448.363548] LustreError: 149028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4448.778509] Lustre: server umount lustre-MDT0001 complete [ 4459.548731] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 4462.536866] Key type lgssc unregistered [ 4462.787241] LNet: 154663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4462.792965] LNetError: 154663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4462.808518] LNet: Removed LNI 192.168.203.135@tcp [ 4463.633928] Key type .llcrypt unregistered [ 4463.642306] Key type ._llcrypt unregistered [ 4499.323397] Key type ._llcrypt registered [ 4499.325972] Key type .llcrypt registered [ 4499.446096] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 4500.380215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4500.508538] alg: No test for adler32 (adler32-zlib) [ 4501.692271] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4501.954791] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 4503.696273] Key type lgssc registered [ 4505.120962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4513.371393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4513.388462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.957795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.259513] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4524.515923] LustreError: 156184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.540375] LustreError: 156184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4528.395736] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4529.637457] LustreError: 156185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.114136] LustreError: 156213:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4532.150227] LustreError: 156213:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4532.169072] LustreError: 156213:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4532.815679] Lustre: server umount lustre-MDT0000 complete [ 4553.410902] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 4564.373201] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 4600.241619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4600.563050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4600.603333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4600.691281] Lustre: lustre-MDT0000: new disk, initializing [ 4600.752178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.769395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4605.661132] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4616.893251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.025776] Lustre: 159519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4617.085630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4617.090810] Lustre: Skipped 1 previous similar message [ 4617.158515] Lustre: lustre-MDT0001: new disk, initializing [ 4617.245151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4617.276462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4617.282909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4621.778463] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4631.830863] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4639.705772] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4650.572535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4651.170171] Lustre: lustre-OST0000: new disk, initializing [ 4651.183906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4651.213119] Lustre: 161486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4651.463365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4657.864718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4657.894291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4658.085586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4660.761244] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 4671.962654] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4677.476716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4679.514278] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:26:57 (1788445617) [ 4919.266709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.267196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4919.270686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4919.285421] Lustre: Skipped 2 previous similar messages [ 4919.309026] Lustre: Skipped 2 previous similar messages [ 4924.384164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4924.399434] Lustre: Skipped 2 previous similar messages [ 4929.505753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4929.522330] Lustre: Skipped 2 previous similar messages [ 4932.064470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4932.369920] Lustre: server umount lustre-MDT0000 complete [ 4934.623887] LustreError: 161417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.638606] LustreError: 161417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4936.112841] LustreError: 159511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445876 with bad export cookie 2753768137759234019 [ 4936.125542] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.134163] LustreError: 159511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4939.752141] LustreError: 159524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4939.756485] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.827741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4942.515816] Lustre: server umount lustre-MDT0001 complete [ 4952.722330] Lustre: server umount lustre-OST0000 complete [ 4961.586239] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 4972.594738] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 5012.252209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.472760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5012.494553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5012.555369] Lustre: lustre-MDT0000: new disk, initializing [ 5012.629263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5012.669479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5016.865981] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5026.829502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.934374] Lustre: 166148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5027.006443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5027.010340] Lustre: Skipped 1 previous similar message [ 5027.081372] Lustre: lustre-MDT0001: new disk, initializing [ 5027.191271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5027.229500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5027.238848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5031.755442] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5041.639492] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5047.412608] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5053.540693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.795183] Lustre: lustre-OST0000: new disk, initializing [ 5053.798853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5053.804469] Lustre: 168116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5053.874060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.185087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5060.212834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5060.257881] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5060.336205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5071.755487] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5077.251207] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5077.452772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5082.271498] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5082.598545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5086.177800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5086.178193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.188378] LustreError: Skipped 1 previous similar message [ 5086.190079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.190086] Lustre: Skipped 1 previous similar message [ 5086.260900] Lustre: Skipped 1 previous similar message [ 5090.803500] Lustre: server umount lustre-OST0000 complete [ 5096.418579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5096.419882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.424662] Lustre: Skipped 1 previous similar message [ 5096.429169] Lustre: Skipped 1 previous similar message [ 5100.835827] Lustre: server umount lustre-MDT0000 complete [ 5101.536349] LustreError: 167312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.559352] LustreError: 167312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5104.491126] LustreError: 166140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446044 with bad export cookie 2753768137759392849 [ 5104.495437] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5104.500880] LustreError: 166140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5104.767980] Lustre: server umount lustre-MDT0001 complete [ 5113.407867] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:34:11 (1788446051) [ 5131.900680] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 5143.097792] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 5183.410850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.680807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5183.713915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5183.811692] Lustre: lustre-MDT0000: new disk, initializing [ 5183.972114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.991627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5188.816689] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5199.820940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.956261] Lustre: 172873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5200.304654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5205.285942] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5214.082611] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5220.124048] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5228.040526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5228.371646] Lustre: lustre-OST0000: new disk, initializing [ 5228.374919] Lustre: Skipped 1 previous similar message [ 5228.391201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5228.396182] Lustre: Skipped 2 previous similar messages [ 5228.401294] Lustre: 174851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5234.289494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5234.305676] Lustre: Skipped 1 previous similar message [ 5234.317225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5234.463488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5235.794858] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5246.801601] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5255.136966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.138598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.157576] Lustre: Skipped 1 previous similar message [ 5255.188835] Lustre: Skipped 1 previous similar message [ 5261.046867] Lustre: server umount lustre-OST0000 complete [ 5265.376565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.570265] Lustre: server umount lustre-MDT0000 complete [ 5275.494708] LustreError: 175958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446215 with bad export cookie 2753768137759393724 [ 5275.496546] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.505057] LustreError: 175958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5275.888199] Lustre: server umount lustre-MDT0001 complete [ 5284.892690] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:37:03 (1788446223) [ 5301.820123] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 5312.926605] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 5347.907406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.133651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5348.156324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5348.272093] Lustre: lustre-MDT0000: new disk, initializing [ 5348.382065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.388171] Lustre: Skipped 2 previous similar messages [ 5348.403637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5353.034287] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5363.533933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.684740] Lustre: 179178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5364.091707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5368.751709] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5377.844656] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5383.379446] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5389.892766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5390.123610] Lustre: 181155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5395.840357] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5398.069737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5398.169138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5404.953475] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5413.355322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.366723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.390945] Lustre: Skipped 1 previous similar message [ 5413.398898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.409492] Lustre: Skipped 6 previous similar messages [ 5417.817477] Lustre: server umount lustre-OST0000 complete [ 5427.589083] Lustre: server umount lustre-MDT0000 complete [ 5429.217800] LustreError: 179190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.242440] LustreError: 179190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5431.439277] LustreError: 179171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446371 with bad export cookie 2753768137759394809 [ 5431.439884] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.449942] LustreError: 179171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5431.837062] Lustre: server umount lustre-MDT0001 complete [ 5445.444559] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 5449.276642] Key type lgssc unregistered [ 5449.726397] LNet: 183012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5449.750379] LNetError: 183012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5449.794449] LNet: Removed LNI 192.168.203.135@tcp [ 5451.141808] Key type .llcrypt unregistered [ 5451.145475] Key type ._llcrypt unregistered [ 5474.474628] Key type ._llcrypt registered [ 5474.477804] Key type .llcrypt registered [ 5474.631746] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 5475.595618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5475.635601] alg: No test for adler32 (adler32-zlib) [ 5476.726234] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5476.985805] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 5478.695171] Key type lgssc registered [ 5479.723798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.812244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.827103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5488.441177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.401786] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5498.852333] LustreError: 184444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.894329] LustreError: 184444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5501.023603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.448684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5505.819542] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5513.166182] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5518.815501] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5524.229164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5524.522812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.863665] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5533.759548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5538.325412] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5557.218915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5557.225731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.236935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5559.272742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.293817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5559.299116] Lustre: Skipped 1 previous similar message [ 5561.688024] Lustre: server umount lustre-OST0000 complete [ 5567.968346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5567.976490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.992071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5571.311496] Lustre: server umount lustre-MDT0000 complete [ 5574.591560] LustreError: 184425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446514 with bad export cookie 8047897917702568605 [ 5574.597296] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5574.604460] LustreError: 184425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5574.973805] Lustre: server umount lustre-MDT0001 complete [ 5585.278837] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 5587.796520] Key type lgssc unregistered [ 5588.049418] LNet: 188003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5588.056668] LNetError: 188003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5588.071862] LNet: Removed LNI 192.168.203.135@tcp [ 5588.947284] Key type .llcrypt unregistered [ 5588.956648] Key type ._llcrypt unregistered [ 5612.171699] Key type ._llcrypt registered [ 5612.175571] Key type .llcrypt registered [ 5612.284599] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 5613.364900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5613.417922] alg: No test for adler32 (adler32-zlib) [ 5614.531160] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5614.791916] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 5616.519171] Key type lgssc registered [ 5617.718916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5626.626582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5626.657580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.139979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.044448] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5637.613194] LustreError: 189423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5637.633026] LustreError: 189423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5641.946993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.348751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.585779] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5653.451938] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5658.985300] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5665.513932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.883326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.248789] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5673.094456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5679.306745] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5687.422646] Lustre: lustre-OST0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 5688.805281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5688.808565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.815755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.836193] Lustre: Skipped 1 previous similar message [ 5692.141343] Lustre: server umount lustre-OST0000 complete [ 5699.121349] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8888d8bb4800.ost_server_uuid 50 [ 5700.749729] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5713.890544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5713.902766] LustreError: Skipped 1 previous similar message [ 5713.906452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.916370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.922990] Lustre: Skipped 1 previous similar message [ 5718.499468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.502913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.521231] Lustre: Skipped 1 previous similar message [ 5722.141589] Lustre: server umount lustre-MDT0000 complete [ 5723.616724] LustreError: 189424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5723.635077] LustreError: 189424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5725.396590] LustreError: 189403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446665 with bad export cookie 17555363713971051000 [ 5725.402321] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5725.414108] LustreError: 189403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5725.702083] Lustre: server umount lustre-MDT0001 complete [ 5733.559941] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:44:31 (1788446671) [ 5739.904281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.226079] LustreError: 193337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.260464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.733703] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5745.635752] LustreError: 193338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.769429] LustreError: 193337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.917583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.455070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5760.912543] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5769.727923] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5775.545535] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5782.113948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.580120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.654448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5789.048344] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5795.224612] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5801.494293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5801.702269] Lustre: lustre-OST0001: new disk, initializing [ 5801.706111] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5801.711584] Lustre: 196177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5801.776649] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5806.959905] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5808.708295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5808.716195] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5808.791982] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5815.551706] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5819.373932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.391938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.403865] Lustre: Skipped 1 previous similar message [ 5824.483758] Lustre: server umount lustre-OST0000 complete [ 5824.506073] LustreError: 195161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5824.521416] LustreError: 195161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5830.279671] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5830.506918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5834.801068] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5834.989094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5842.024795] LustreError: 195161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.040851] LustreError: 195161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5850.089247] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5850.093804] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5850.097714] Lustre: Skipped 1 previous similar message [ 5850.098778] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5850.110378] LustreError: Skipped 1 previous similar message [ 5850.132490] Lustre: Skipped 2 previous similar messages [ 5853.721762] Lustre: server umount lustre-OST0001 complete [ 5857.760746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5860.322844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.336485] Lustre: Skipped 2 previous similar messages [ 5863.361761] Lustre: server umount lustre-MDT0000 complete [ 5866.767067] LustreError: 193318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446806 with bad export cookie 17555363713971052071 [ 5866.772078] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5866.786742] LustreError: 193318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5867.074699] Lustre: server umount lustre-MDT0001 complete [ 5889.373632] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:47:07 (1788446827) [ 5895.304646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5895.609364] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5895.669492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5895.839720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5900.832342] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5909.595984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.810669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5909.854933] Lustre: 199614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5914.887964] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5922.939274] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5928.315845] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5934.794056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5934.984911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5941.134784] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5943.517220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5949.058805] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5956.188540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.348164] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5963.097802] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 5965.523436] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5970.288149] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5974.496610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5974.501410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.504843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5974.510196] Lustre: Skipped 1 previous similar message [ 5974.512690] Lustre: Skipped 3 previous similar messages [ 5978.849314] Lustre: server umount lustre-OST0000 complete [ 5981.154316] LustreError: 201464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.175138] LustreError: 201464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5996.512861] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.525592] LustreError: Skipped 1 previous similar message [ 5999.947707] Lustre: server umount lustre-OST0001 complete [ 6006.754918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.769452] Lustre: Skipped 2 previous similar messages [ 6006.777573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.786496] Lustre: Skipped 7 previous similar messages [ 6009.694421] Lustre: server umount lustre-MDT0000 complete [ 6012.920793] LustreError: 199605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446952 with bad export cookie 17555363713971053478 [ 6012.927437] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6012.933572] LustreError: 199605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6013.239258] Lustre: server umount lustre-MDT0001 complete [ 6037.735391] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:49:36 (1788446976) [ 6052.079677] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 6061.398765] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 6093.326399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.536185] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6093.561325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6093.620246] Lustre: lustre-MDT0000: new disk, initializing [ 6093.711123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.715548] Lustre: Skipped 3 previous similar messages [ 6093.727499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6097.545893] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6106.979108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.117449] Lustre: 207720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6107.161726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6107.166716] Lustre: Skipped 1 previous similar message [ 6107.241961] Lustre: lustre-MDT0001: new disk, initializing [ 6107.344439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6107.349218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6111.528755] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6120.213924] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6125.243666] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6131.981732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6132.199532] Lustre: lustre-OST0000: new disk, initializing [ 6132.203314] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6132.208717] Lustre: 209688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6137.890332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6137.910532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6138.059788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6138.303576] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6146.693361] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6150.533706] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6150.741650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6154.916554] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6155.127521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6158.824942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6158.828936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.839096] LustreError: Skipped 2 previous similar messages [ 6158.842372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6158.851859] Lustre: Skipped 2 previous similar messages [ 6158.872183] Lustre: Skipped 1 previous similar message [ 6163.231300] Lustre: server umount lustre-OST0000 complete [ 6172.760827] Lustre: server umount lustre-MDT0000 complete [ 6174.177667] LustreError: 207732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.196392] LustreError: 207732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6175.992304] LustreError: 207713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447115 with bad export cookie 17555363713971055256 [ 6176.002535] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6176.004578] LustreError: 207713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6176.298248] Lustre: server umount lustre-MDT0001 complete [ 6184.469667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.332628] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6197.135277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.958745] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6207.857828] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6212.823767] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6219.561603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6227.109828] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6235.501087] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6239.349940] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6239.575476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6243.623450] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6243.936552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6247.903993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.919415] LustreError: Skipped 1 previous similar message [ 6260.191183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6260.385951] Lustre: server umount lustre-OST0000 complete [ 6268.194070] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6273.353945] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6301.693460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6302.033783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.039208] Lustre: Skipped 5 previous similar messages [ 6307.949747] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6315.373061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.379749] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6315.392714] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.394231] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6321.183706] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6325.732965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6325.739389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.748097] LustreError: Skipped 2 previous similar messages [ 6325.748815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6325.773687] Lustre: Skipped 5 previous similar messages [ 6325.781739] Lustre: Skipped 9 previous similar messages [ 6331.781303] Lustre: server umount lustre-OST0000 complete [ 6341.899476] Lustre: server umount lustre-MDT0000 complete [ 6345.612422] LustreError: 211888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447285 with bad export cookie 17555363713971056145 [ 6345.622231] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6345.626388] LustreError: 211888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6345.878161] Lustre: server umount lustre-MDT0001 complete [ 6353.363794] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:54:51 (1788447291) [ 6359.541122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.145611] LustreError: 217876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.191914] LustreError: 217876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6364.676899] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6372.417133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.240134] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6384.190477] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6389.588504] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6395.954416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6401.460401] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6408.841405] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.741425] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6415.189269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6422.054710] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6422.466924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6430.230613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6430.420813] Lustre: lustre-OST0001: new disk, initializing [ 6430.427794] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6430.433315] Lustre: 221247:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6436.256873] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6438.505944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6438.526038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6438.629963] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6445.176114] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6449.529553] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6449.797815] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6454.660769] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6454.919484] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6459.370987] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.385531] LustreError: Skipped 2 previous similar messages [ 6471.135447] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6471.250926] Lustre: server umount lustre-OST0001 complete [ 6477.512270] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6477.738665] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6482.253325] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6482.526309] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6513.158600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6518.774615] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6518.792379] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6518.797414] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6518.845894] Lustre: Skipped 2 previous similar messages [ 6521.121915] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6528.348376] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6530.223832] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6538.801178] Lustre: server umount lustre-OST0001 complete [ 6562.160569] Lustre: server umount lustre-OST0000 complete [ 6572.050404] Lustre: server umount lustre-MDT0000 complete [ 6575.526713] LustreError: 219721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447515 with bad export cookie 17555363713971057104 [ 6575.534108] LustreError: 219721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6575.542804] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6581.890271] Lustre: server umount lustre-MDT0001 complete [ 6603.321620] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:59:01 (1788447541) [ 6608.916266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.100638] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6609.120216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6609.272495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6609.282518] Lustre: Skipped 5 previous similar messages [ 6613.456289] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6621.597530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.717972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6621.758920] Lustre: 226491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6627.704412] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6635.298572] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6640.592258] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.274186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.430586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6652.243275] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6658.389910] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6666.808568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.856740] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6672.444789] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6679.249858] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.436320] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6684.625312] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6684.867683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6689.743710] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6689.927645] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6695.031682] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8888d8bb7800.ost_server_uuid 50 [ 6696.689453] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8888d8bb7800.ost_server_uuid in IDLE state after 0 sec [ 6697.810864] Lustre: Permanently deactivating lustre-OST0001 [ 6697.824878] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6701.457211] Lustre: Permanently reactivating lustre-OST0001 [ 6705.638078] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.656515] Lustre: Skipped 10 previous similar messages [ 6705.674929] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6705.679264] Lustre: Skipped 21 previous similar messages [ 6711.160131] Lustre: server umount lustre-OST0001 complete [ 6711.268681] LustreError: 229022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.290584] LustreError: 229022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 6715.872894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6715.879381] LustreError: Skipped 5 previous similar messages [ 6720.815599] Lustre: server umount lustre-OST0000 complete [ 6730.047919] Lustre: server umount lustre-MDT0000 complete [ 6733.160919] LustreError: 228353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447673 with bad export cookie 17555363713971058399 [ 6733.164446] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6733.173908] LustreError: 228353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6733.548703] Lustre: server umount lustre-MDT0001 complete [ 6756.994400] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:01:35 (1788447695) [ 6764.313208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6764.579193] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6764.610798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6764.620368] Lustre: Skipped 5 previous similar messages [ 6769.730994] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6778.229235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.309789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6778.351908] Lustre: 233196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6783.332329] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6790.233844] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6794.474638] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6800.717218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.963147] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6801.003562] Lustre: Permanently deactivating lustre-OST0000 [ 6801.043928] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6807.904908] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6811.187707] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6815.160427] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6820.940694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6820.997953] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6826.182560] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6827.676335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6832.476969] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.547065] Lustre: Permanently reactivating lustre-OST0000 [ 6834.559896] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6834.564730] Lustre: Skipped 2 previous similar messages [ 6835.575311] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6835.582987] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6835.608557] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6835.680382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6835.721562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6852.466324] Lustre: server umount lustre-OST0001 complete [ 6862.652391] Lustre: server umount lustre-OST0000 complete [ 6872.468799] Lustre: server umount lustre-MDT0000 complete [ 6875.683550] LustreError: 236074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447815 with bad export cookie 17555363713971060072 [ 6875.687074] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6875.702698] LustreError: 236074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6875.995745] Lustre: server umount lustre-MDT0001 complete [ 6884.481814] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 6886.512824] Key type lgssc unregistered [ 6886.728219] LNet: 238081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6886.733741] LNetError: 238081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6886.755738] LNet: Removed LNI 192.168.203.135@tcp [ 6887.457640] Key type .llcrypt unregistered [ 6887.459814] Key type ._llcrypt unregistered [ 6911.228681] Key type ._llcrypt registered [ 6911.231967] Key type .llcrypt registered [ 6911.363877] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 6912.246422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6912.264394] alg: No test for adler32 (adler32-zlib) [ 6913.402036] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6913.745832] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 6915.415144] Key type lgssc registered [ 6916.481974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6927.071985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6927.103647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.732350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.221379] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6939.105859] LustreError: 239546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.138949] LustreError: 239546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6942.976499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.097241] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6943.131709] Lustre: 239532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6943.150412] Lustre: Permanently deactivating lustre-MDT0001 [ 6943.175944] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6943.612802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6944.667636] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6944.675159] LustreError: 239576:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6944.694154] LustreError: 239576:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6944.705579] LustreError: 239576:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6947.213537] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6953.912419] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6959.276797] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6965.666580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6966.109293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.178787] LustreError: 241387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.196181] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6972.086578] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6973.410967] LustreError: 241533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.459249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6978.738596] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6979.555290] LustreError: 241390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.575621] LustreError: 241390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6984.678519] LustreError: 241533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.695967] LustreError: 241533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6985.511378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.864889] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6991.351342] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6991.354507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6991.781619] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 6998.330663] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7000.769884] Lustre: Permanently reactivating lustre-MDT0001 [ 7000.784162] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7000.788925] Lustre: Skipped 2 previous similar messages [ 7007.352306] Lustre: lustre-MDT0001: Client 399f9c23-37ff-48a4-b3ac-836f213c298e (at 192.168.203.35@tcp) reconnecting [ 7013.056485] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7014.757404] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7020.086575] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7020.212482] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7025.418796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7025.630989] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7028.093137] Lustre: Permanently deactivating lustre-MDT0001 [ 7028.117994] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7028.133955] Lustre: Skipped 2 previous similar messages [ 7038.113290] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7046.113748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7046.128547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.146260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7047.649679] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.652720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7047.676853] Lustre: Skipped 1 previous similar message [ 7047.696789] Lustre: Skipped 3 previous similar messages [ 7052.769655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7057.894198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7057.907018] Lustre: Skipped 6 previous similar messages [ 7058.405961] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7058.652235] Lustre: server umount lustre-MDT0000 complete [ 7063.009745] LustreError: 243989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.049242] LustreError: 243989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7063.665810] LustreError: 239527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448003 with bad export cookie 3779374509401469355 [ 7063.684439] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7063.690662] LustreError: 239527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7064.782354] Lustre: server umount lustre-MDT0001 complete [ 7070.343398] Lustre: server umount lustre-OST0000 complete [ 7081.636785] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:07:00 (1788448020) [ 7091.636646] Lustre: server umount lustre-OST0001 complete [ 7097.484613] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 7106.403718] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7133.047192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7133.227433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7133.236922] Lustre: Skipped 2 previous similar messages [ 7133.254074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7133.347263] Lustre: lustre-MDT0000: new disk, initializing [ 7133.440582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.455756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7137.561090] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7146.842651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.949136] Lustre: 247822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7146.995371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7147.005590] Lustre: Skipped 1 previous similar message [ 7147.095277] Lustre: lustre-MDT0001: new disk, initializing [ 7147.168795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7147.210121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7147.221532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7150.561873] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7159.580834] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7165.921915] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7173.294731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7173.846172] Lustre: lustre-OST0000: new disk, initializing [ 7173.853210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7173.863411] Lustre: 249795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7173.991440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.195956] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7182.438165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7182.454794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7182.568135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7193.616784] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7196.364927] LustreError: 247831:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7205.858912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7206.019525] Lustre: lustre-OST0001: new disk, initializing [ 7206.022940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7206.028885] Lustre: 250908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7206.132143] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7206.439062] LustreError: 247831:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7206.463315] LustreError: 247829:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7211.844773] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7212.204060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7212.209969] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7212.370446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7216.559118] LustreError: 247829:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7221.451349] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7223.396395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7227.881753] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7227.886588] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7227.900081] Lustre: Skipped 1 previous similar message [ 7227.910515] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7227.917836] LustreError: Skipped 1 previous similar message [ 7227.931737] Lustre: Skipped 1 previous similar message [ 7231.546065] Lustre: server umount lustre-OST0001 complete [ 7232.996678] LustreError: 249779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.014616] LustreError: 249779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7248.362953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.377594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7248.381780] Lustre: Skipped 2 previous similar messages [ 7248.394315] Lustre: Skipped 1 previous similar message [ 7254.427631] Lustre: server umount lustre-OST0000 complete [ 7260.132946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7260.149439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.144434] Lustre: server umount lustre-MDT0000 complete [ 7268.833966] LustreError: 250562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7268.868404] LustreError: 250562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7270.258425] LustreError: 248620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448210 with bad export cookie 3779374509401472169 [ 7270.261954] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.271241] LustreError: 248620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7276.870974] Lustre: server umount lustre-MDT0001 complete [ 7287.525298] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7290.668746] Key type lgssc unregistered [ 7291.027042] LNet: 252949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7291.047721] LNetError: 252949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7291.082273] LNet: Removed LNI 192.168.203.135@tcp [ 7292.056987] Key type .llcrypt unregistered [ 7292.060679] Key type ._llcrypt unregistered [ 7332.465229] Key type ._llcrypt registered [ 7332.468494] Key type .llcrypt registered [ 7332.639500] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7333.877827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7333.901057] alg: No test for adler32 (adler32-zlib) [ 7334.885815] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7335.137287] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7336.833683] Key type lgssc registered [ 7338.130091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7345.838486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7345.884156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.192719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7347.228502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7347.355280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7351.967305] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7359.997865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7360.077172] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7360.093578] Lustre: 255157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7360.470145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7365.809261] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7375.143016] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7381.259237] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7387.065173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7387.250758] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7387.414484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.114300] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7396.046444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7400.826255] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7457.251411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7457.269411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.296829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7457.760387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7457.776842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7457.806083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7459.927470] Lustre: server umount lustre-OST0000 complete [ 7464.511592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7477.095549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.457227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.862951] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7482.879832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7482.883026] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7482.923985] Lustre: Skipped 1 previous similar message [ 7484.442606] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7492.544879] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7534.058601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.075952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7534.076813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7534.088301] Lustre: Skipped 1 previous similar message [ 7539.238183] Lustre: server umount lustre-OST0000 complete [ 7544.292376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7544.300636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7544.304285] Lustre: Skipped 1 previous similar message [ 7544.316893] Lustre: Skipped 2 previous similar messages [ 7549.417710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7549.424106] Lustre: Skipped 1 previous similar message [ 7550.340713] Lustre: server umount lustre-MDT0000 complete [ 7554.534369] LustreError: 255168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7554.560287] LustreError: 255168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7554.891379] LustreError: 255899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448494 with bad export cookie 5317470926859436492 [ 7554.891942] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7554.911562] LustreError: 255899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7555.141929] Lustre: server umount lustre-MDT0001 complete [ 7567.001637] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7569.417791] Key type lgssc unregistered [ 7569.754422] LNet: 260583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7569.760641] LNetError: 260583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7569.792903] LNet: Removed LNI 192.168.203.135@tcp [ 7570.697679] Key type .llcrypt unregistered [ 7570.700282] Key type ._llcrypt unregistered [ 7595.571282] Key type ._llcrypt registered [ 7595.577078] Key type .llcrypt registered [ 7595.696160] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7597.065546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7597.094158] alg: No test for adler32 (adler32-zlib) [ 7598.215037] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7598.514712] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7600.207171] Key type lgssc registered [ 7601.418762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7609.285369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7609.323280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.640279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.717156] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7621.092018] LustreError: 262002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.119256] LustreError: 262002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7622.757097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7623.216867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7628.420609] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7636.188233] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7642.206979] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7648.733678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7649.136912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7655.318364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7655.401316] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7662.491355] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7689.194091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7689.219813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7692.460669] Lustre: server umount lustre-OST0000 complete [ 7699.428800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.433609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7699.442425] Lustre: Skipped 2 previous similar messages [ 7699.468599] Lustre: Skipped 2 previous similar messages [ 7702.329950] Lustre: server umount lustre-MDT0000 complete [ 7704.545068] LustreError: 262002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7704.566467] LustreError: 262002:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7705.755218] LustreError: 261981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448645 with bad export cookie 121080403634425664 [ 7705.770592] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.088313] Lustre: server umount lustre-MDT0001 complete [ 7715.841878] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7718.430793] Key type lgssc unregistered [ 7718.767373] LNet: 266308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7718.775378] LNetError: 266308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7718.793804] LNet: Removed LNI 192.168.203.135@tcp [ 7719.677546] Key type .llcrypt unregistered [ 7719.681061] Key type ._llcrypt unregistered [ 7736.400480] Key type ._llcrypt registered [ 7736.402591] Key type .llcrypt registered [ 7736.516544] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7737.867866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7737.887140] alg: No test for adler32 (adler32-zlib) [ 7738.947145] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7739.183399] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7740.847190] Key type lgssc registered [ 7742.030752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7749.426157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7749.443086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.869875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.907736] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7760.358970] LustreError: 267532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.377207] LustreError: 267532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7763.714082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.247448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7769.051580] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7776.175529] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7780.976983] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7787.281239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.808505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.461628] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7796.042345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7803.478752] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7815.650808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7815.655932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.665155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7816.680667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.682211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7816.699805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7816.706970] Lustre: Skipped 1 previous similar message [ 7819.877268] Lustre: server umount lustre-OST0000 complete [ 7825.889546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7825.902359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.915785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7829.969927] Lustre: server umount lustre-MDT0000 complete [ 7832.034828] LustreError: 267533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.065385] LustreError: 267533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7834.189289] LustreError: 267514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448774 with bad export cookie 12391155328887770781 [ 7834.195901] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7834.203809] LustreError: 267514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7834.577636] Lustre: server umount lustre-MDT0001 complete [ 7844.383664] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7846.667659] Key type lgssc unregistered [ 7846.952921] LNet: 271213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7846.962207] LNetError: 271213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7846.987099] LNet: Removed LNI 192.168.203.135@tcp [ 7847.730946] Key type .llcrypt unregistered [ 7847.737677] Key type ._llcrypt unregistered [ 7868.087921] Key type ._llcrypt registered [ 7868.090938] Key type .llcrypt registered [ 7868.182649] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 7869.012620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.036492] alg: No test for adler32 (adler32-zlib) [ 7870.036884] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7870.270274] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 7871.935168] Key type lgssc registered [ 7872.817639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7879.774987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7879.800608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7881.269512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7885.099817] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7891.426543] LustreError: 272621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.449080] LustreError: 272621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7892.277102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.611313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7896.350321] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7903.447711] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7908.125407] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7914.333502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7914.774066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7920.197224] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 7926.705929] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7927.094779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7948.769430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7948.782803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.813097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.867274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.870535] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7952.871589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.884839] Lustre: Skipped 1 previous similar message [ 7954.204797] Lustre: server umount lustre-OST0000 complete [ 7957.985975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.992918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7958.017995] Lustre: Skipped 1 previous similar message [ 7958.035213] Lustre: Skipped 1 previous similar message [ 7963.107132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7963.112165] Lustre: Skipped 1 previous similar message [ 7963.978718] Lustre: server umount lustre-MDT0000 complete [ 7967.235920] LustreError: 272602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448907 with bad export cookie 7825572339727183843 [ 7967.247645] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7967.253713] LustreError: 272602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7967.579401] Lustre: server umount lustre-MDT0001 complete [ 7978.040804] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 7980.461744] Key type lgssc unregistered [ 7980.718508] LNet: 276930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7980.724184] LNetError: 276930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7980.737846] LNet: Removed LNI 192.168.203.135@tcp [ 7981.619838] Key type .llcrypt unregistered [ 7981.622452] Key type ._llcrypt unregistered [ 7999.028772] Key type ._llcrypt registered [ 7999.031287] Key type .llcrypt registered [ 7999.167907] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8000.125676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8000.138649] alg: No test for adler32 (adler32-zlib) [ 8001.250693] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8001.507974] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 8003.231198] Key type lgssc registered [ 8004.400347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8011.166043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8011.185767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.741004] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.481447] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8023.010954] LustreError: 278166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.032122] LustreError: 278166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8024.922561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.197518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8029.849648] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8038.318355] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8044.696097] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8052.388780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8052.858854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.986304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8059.293450] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8067.160347] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8080.865755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8080.871640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.875672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8080.897943] Lustre: Skipped 1 previous similar message [ 8085.990887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8085.999354] Lustre: Skipped 1 previous similar message [ 8086.860636] Lustre: server umount lustre-OST0000 complete [ 8091.106692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.124043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8091.135797] Lustre: Skipped 1 previous similar message [ 8091.150162] Lustre: Skipped 1 previous similar message [ 8096.233707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8096.243801] Lustre: Skipped 1 previous similar message [ 8097.149536] Lustre: server umount lustre-MDT0000 complete [ 8101.064769] LustreError: 278131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449040 with bad export cookie 2674901973309362788 [ 8101.069704] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8101.074118] LustreError: 278131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8101.588481] Lustre: server umount lustre-MDT0001 complete [ 8112.577571] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 8115.222799] Key type lgssc unregistered [ 8115.721643] LNet: 281872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8115.751535] LNetError: 281872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8115.779969] LNet: Removed LNI 192.168.203.135@tcp [ 8116.967935] Key type .llcrypt unregistered [ 8116.970315] Key type ._llcrypt unregistered [ 8131.163892] Key type ._llcrypt registered [ 8131.168371] Key type .llcrypt registered [ 8131.328184] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8150.709341] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8166.476289] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8167.581471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8167.649627] alg: No test for adler32 (adler32-zlib) [ 8168.764905] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8169.002358] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 8170.679275] Key type lgssc registered [ 8171.596282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.613634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8207.651685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.788374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8208.821566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8208.947697] Lustre: lustre-MDT0000: new disk, initializing [ 8209.027580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.067126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8213.622041] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8224.521475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.734320] Lustre: 285425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8224.801953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8224.806046] Lustre: Skipped 1 previous similar message [ 8224.968735] Lustre: lustre-MDT0001: new disk, initializing [ 8225.101993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.151640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8225.176460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8230.845250] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8240.774823] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8246.073877] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.457468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.779251] Lustre: lustre-OST0000: new disk, initializing [ 8252.782847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8252.787612] Lustre: 287394:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8252.874261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.805026] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8259.146832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8259.158593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8259.310719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8268.407670] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8273.648936] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8273.849929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8278.718483] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8278.943425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8285.164547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.164996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8285.197601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8285.203818] Lustre: Skipped 1 previous similar message [ 8290.274303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8290.280118] Lustre: Skipped 1 previous similar message [ 8295.391121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8295.405291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8295.408643] Lustre: Skipped 1 previous similar message [ 8295.538450] Lustre: server umount lustre-OST0000 complete [ 8300.522108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.526888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.532613] Lustre: Skipped 2 previous similar messages [ 8305.635409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.646394] Lustre: Skipped 2 previous similar messages [ 8313.311231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8313.613619] Lustre: server umount lustre-MDT0000 complete [ 8317.505996] LustreError: 285419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449257 with bad export cookie 2347598206428425044 [ 8317.514276] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.517445] LustreError: 285419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8317.844755] Lustre: server umount lustre-MDT0001 complete [ 8326.439535] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:27:44 (1788449264) [ 8333.461709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.179149] LustreError: 289809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.203090] LustreError: 289809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8334.301299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.382222] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8339.427242] LustreError: 289810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.546161] LustreError: 289809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.827579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.265016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8353.224550] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8359.675319] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8364.488276] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8370.304434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8370.619150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.900039] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8384.271809] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8394.209386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8394.215437] LustreError: Skipped 1 previous similar message [ 8394.223266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8394.246084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8394.255293] Lustre: Skipped 2 previous similar messages [ 8395.247278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8398.712483] Lustre: server umount lustre-OST0000 complete [ 8404.450758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8404.467447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8404.483554] Lustre: Skipped 1 previous similar message [ 8408.501433] Lustre: server umount lustre-MDT0000 complete [ 8410.592414] LustreError: 292494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.619428] LustreError: 292494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8411.974395] LustreError: 289789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449351 with bad export cookie 2347598206428425926 [ 8411.982906] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8411.987134] LustreError: 289789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8412.275643] Lustre: server umount lustre-MDT0001 complete [ 8421.212670] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 8423.786201] Key type lgssc unregistered [ 8423.991858] LNet: 293347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8424.006183] LNetError: 293347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8424.023328] LNet: Removed LNI 192.168.203.135@tcp [ 8424.952217] Key type .llcrypt unregistered [ 8424.954328] Key type ._llcrypt unregistered [ 8450.242951] Key type ._llcrypt registered [ 8450.244705] Key type .llcrypt registered [ 8450.371947] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8468.429513] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8469.897418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8470.048025] alg: No test for adler32 (adler32-zlib) [ 8471.407782] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8471.947618] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 8473.711369] Key type lgssc registered [ 8475.014459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8522.568498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8522.590882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.902599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8523.925146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8523.994779] Lustre: lustre-MDT0000: new disk, initializing [ 8524.138458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.148348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8528.673182] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8538.406092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.611702] Lustre: 297177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8538.711923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8538.718264] Lustre: Skipped 1 previous similar message [ 8538.850644] Lustre: lustre-MDT0001: new disk, initializing [ 8538.990666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8539.066801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8539.071000] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8543.125375] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8552.312708] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8558.536442] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8565.681526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8565.892447] Lustre: lustre-OST2710: new disk, initializing [ 8565.896236] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8565.901231] Lustre: 299145:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8565.999714] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8572.229658] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8575.563311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8575.573823] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8575.695115] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8581.605826] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.663723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.853767] Lustre: lustre-OST03e8: new disk, initializing [ 8587.857597] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8587.863430] Lustre: 300209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8587.932967] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8593.769540] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8597.621853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8597.644512] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8597.781546] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8600.877978] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8608.395444] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8608.587538] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8612.891581] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8613.142073] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8617.587835] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8617.801847] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8622.608636] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8622.834550] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8628.710520] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.718237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8628.725612] Lustre: Skipped 1 previous similar message [ 8628.741101] Lustre: Skipped 3 previous similar messages [ 8629.839551] Lustre: server umount lustre-MDT0000 complete [ 8633.514232] LustreError: 297169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449573 with bad export cookie 14676061058750850390 [ 8633.529229] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8633.531476] LustreError: 297169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8633.878451] Lustre: server umount lustre-MDT0001 complete [ 8648.129631] Lustre: server umount lustre-OST2710 complete [ 8649.168240] Lustre: 294976:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449573/real 1788449573] req@000000001d694ba9 x1875325129653376/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788449589 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8649.196991] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.212079] Lustre: Skipped 2 previous similar messages [ 8651.739945] Lustre: server umount lustre-OST03e8 complete [ 8671.300324] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 8680.771950] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8716.683668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.865823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8716.894544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8716.991992] Lustre: lustre-MDT0000: new disk, initializing [ 8717.060472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.086467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8721.239855] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8730.773493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8730.889292] Lustre: 305456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8730.935288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8730.938567] Lustre: Skipped 1 previous similar message [ 8731.024709] Lustre: lustre-MDT0001: new disk, initializing [ 8731.207663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8731.268703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8731.288288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8735.372723] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8746.042725] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8752.845171] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8760.106961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8760.294800] Lustre: lustre-OST0000: new disk, initializing [ 8760.299205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8760.306524] Lustre: 307422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8760.382585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.532649] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8769.072831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8769.082096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8769.229204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8776.406198] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8781.252687] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8781.452552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8785.416983] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8785.575463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8789.987255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8789.999355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8790.017364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8790.027953] Lustre: Skipped 1 previous similar message [ 8793.252961] Lustre: server umount lustre-OST0000 complete [ 8797.664210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8797.676258] LustreError: Skipped 1 previous similar message [ 8797.686347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8797.706669] Lustre: Skipped 1 previous similar message [ 8797.717916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.225374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8803.201905] Lustre: server umount lustre-MDT0000 complete [ 8807.453823] LustreError: 307421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449747 with bad export cookie 14676061058750852070 [ 8807.457611] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.467488] LustreError: 307421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8807.689316] Lustre: server umount lustre-MDT0001 complete [ 8817.028745] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:35:54 (1788449754) [ 8818.896408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8820.685611] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:35:58 (1788449758) [ 8824.689599] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8851.515235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.809820] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8851.838889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8851.982082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.834882] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8864.518256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.607950] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8864.633765] Lustre: 311286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8869.658302] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 8891.385674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.767272] Lustre: Denying initial registration attempt from nid 192.168.203.135@tcp, specified as failover [ 8891.781279] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8891.799619] LustreError: 312738:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8891.834462] LustreError: 312738:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8891.844087] LustreError: 312738:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8891.980418] Lustre: server umount lustre-OST0000 complete [ 8891.987707] LustreError: 312738:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8900.578463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8900.588666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.612325] Lustre: Skipped 1 previous similar message [ 8900.618488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.623536] Lustre: Skipped 1 previous similar message [ 8906.196187] Lustre: server umount lustre-MDT0000 complete [ 8908.778062] LustreError: 312032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8908.790557] LustreError: 312032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8910.655161] LustreError: 311277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449850 with bad export cookie 14676061058750852952 [ 8910.669407] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8910.682129] LustreError: 311277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8911.056494] Lustre: server umount lustre-MDT0001 complete [ 8922.862451] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 8925.793245] Key type lgssc unregistered [ 8926.064817] LNet: 313864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8926.080531] LNetError: 313864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8926.106703] LNet: Removed LNI 192.168.203.135@tcp [ 8927.035649] Key type .llcrypt unregistered [ 8927.038473] Key type ._llcrypt unregistered [ 8938.962079] Key type ._llcrypt registered [ 8938.964933] Key type .llcrypt registered [ 8939.105653] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 8940.065338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8940.136286] alg: No test for adler32 (adler32-zlib) [ 8941.291806] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8941.629760] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 8943.375648] Key type lgssc registered [ 8944.580961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8992.061457] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9001.480954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9001.510711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9002.758228] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9002.795990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9003.017986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.083404] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9016.876198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.940792] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9016.968448] Lustre: 316628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9017.204977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9022.107937] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9029.863777] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9035.615995] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9055.158170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9055.534080] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9055.858385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9063.482427] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9069.737522] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9075.170058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9075.193471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.215603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9075.690602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9075.701312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.717834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9077.630754] Lustre: server umount lustre-OST0000 complete [ 9083.874372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9083.885161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.916428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.926569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.931704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.015123] Lustre: server umount lustre-MDT0000 complete [ 9091.372186] LustreError: 316620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450031 with bad export cookie 10114081271568976585 [ 9091.375364] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9091.387778] LustreError: 316620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9091.887368] Lustre: server umount lustre-MDT0001 complete [ 9102.939831] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 9105.807295] Key type lgssc unregistered [ 9106.069649] LNet: 320205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9106.076688] LNetError: 320205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9106.095265] LNet: Removed LNI 192.168.203.135@tcp [ 9107.067961] Key type .llcrypt unregistered [ 9107.071627] Key type ._llcrypt unregistered [ 9130.572596] Key type ._llcrypt registered [ 9130.574586] Key type .llcrypt registered [ 9130.640088] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9131.706936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9131.722442] alg: No test for adler32 (adler32-zlib) [ 9132.784709] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9133.052071] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 9134.760791] Key type lgssc registered [ 9135.990805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9144.429907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9144.442312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.870438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.956401] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9156.071709] LustreError: 321623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9156.091228] LustreError: 321623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9158.943265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9159.449574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9164.648193] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9172.459929] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9177.987397] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9185.957475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9186.384950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.686340] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9201.461463] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9215.456825] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.473653] Lustre: Skipped 1 previous similar message [ 9215.480504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.497737] Lustre: Skipped 2 previous similar messages [ 9218.697728] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [ 9220.583165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.595888] Lustre: Skipped 2 previous similar messages [ 9221.770655] Lustre: server umount lustre-MDT0000 complete [ 9223.797846] LustreError: 323441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.702047] LustreError: 322352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.720521] LustreError: 322352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9225.892941] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450165 with bad export cookie 216777277231344318 [ 9225.900096] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9225.904724] LustreError: 321605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9228.901867] LustreError: 323441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9228.939984] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [ 9230.827706] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.853238] Lustre: Skipped 1 previous similar message [ 9232.502644] Lustre: server umount lustre-MDT0001 complete [ 9239.148586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9248.449649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9251.232868] LustreError: 320933:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ee6ac953 x1875325822738944/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9251.672421] LustreError: 323457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9251.698239] LustreError: 323457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9251.779122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9251.842717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9256.535113] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9260.913731] LustreError: 323458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9260.924735] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9260.936264] LustreError: 323458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9265.352412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.805089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9270.401830] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9270.959490] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9277.470577] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9279.312822] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9284.563848] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9286.211524] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9291.749070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9291.766174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9291.771084] Lustre: Skipped 1 previous similar message [ 9296.369170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9296.375689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.843977] Lustre: server umount lustre-OST0000 complete [ 9301.985407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.008956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9302.018524] Lustre: Skipped 2 previous similar messages [ 9306.594836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9308.057399] Lustre: server umount lustre-MDT0000 complete [ 9311.832807] LustreError: 321606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450251 with bad export cookie 216777277231351920 [ 9311.836961] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9311.846758] LustreError: 321606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9312.189438] Lustre: server umount lustre-MDT0001 complete [ 9323.562544] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 9326.159720] Key type lgssc unregistered [ 9326.452752] LNet: 327913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9326.460845] LNetError: 327913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9326.478951] LNet: Removed LNI 192.168.203.135@tcp [ 9327.352986] Key type .llcrypt unregistered [ 9327.359977] Key type ._llcrypt unregistered [ 9351.706416] Key type ._llcrypt registered [ 9351.718993] Key type .llcrypt registered [ 9351.872128] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9352.752568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9352.941406] alg: No test for adler32 (adler32-zlib) [ 9354.096432] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9354.335042] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 9356.047438] Key type lgssc registered [ 9357.014847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9364.281313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.294330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.662471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9369.877174] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9375.211486] LustreError: 329331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.245080] LustreError: 329331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9377.867430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9378.133273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9382.025516] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9388.213243] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9392.860284] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9395.682821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9395.684641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9395.691755] Lustre: Skipped 1 previous similar message [ 9395.702601] Lustre: Skipped 1 previous similar message [ 9400.805260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9400.821905] Lustre: Skipped 1 previous similar message [ 9401.855492] Lustre: server umount lustre-MDT0000 complete [ 9402.853238] LustreError: 329331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9402.871445] LustreError: 329331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9405.146338] LustreError: 330068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450345 with bad export cookie 1527467438380842512 [ 9405.147830] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9405.162859] LustreError: 330068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9405.619755] Lustre: server umount lustre-MDT0001 complete [ 9417.539158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.797148] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9417.837730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9418.026720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.060680] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9429.506813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9429.583765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9429.616629] Lustre: 331921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9429.900370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9434.377498] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9440.781601] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9446.260446] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9455.475737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9455.658198] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9455.669256] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9455.685895] LustreError: 333802:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9455.705126] LustreError: 333802:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9455.710191] LustreError: 333802:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9455.772298] Lustre: server umount lustre-OST0000 complete [ 9455.778561] LustreError: 333802:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9465.643156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9465.852231] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9465.992346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.245822] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9472.643279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9478.069741] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9485.309045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9485.398830] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9485.501656] Lustre: lustre-OST0001: new disk, initializing [ 9485.505298] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9485.511387] Lustre: 335344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9485.617289] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9491.785603] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9494.635526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9494.649794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9494.783236] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9498.949684] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9505.248381] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9505.251944] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9505.258796] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9505.272347] Lustre: Skipped 1 previous similar message [ 9505.285775] Lustre: Skipped 1 previous similar message [ 9507.674551] Lustre: server umount lustre-OST0001 complete [ 9510.377160] LustreError: 334725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.405545] LustreError: 334725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9513.441917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9513.462701] LustreError: Skipped 1 previous similar message [ 9513.477225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9513.515381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9515.504433] LustreError: 334269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9515.531227] LustreError: 334269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9520.616437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9520.622950] LustreError: 334266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9520.629975] Lustre: Skipped 4 previous similar messages [ 9520.657299] LustreError: 334266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9525.215118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9525.480264] Lustre: server umount lustre-OST0000 complete [ 9530.848560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9530.849246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9530.860251] Lustre: Skipped 2 previous similar messages [ 9530.870817] Lustre: Skipped 1 previous similar message [ 9535.603507] Lustre: server umount lustre-MDT0000 complete [ 9535.977818] LustreError: 331927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9535.995188] LustreError: 331927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9538.968940] LustreError: 331912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450478 with bad export cookie 1527467438380843058 [ 9538.978334] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.980051] LustreError: 331912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9539.369536] Lustre: server umount lustre-MDT0001 complete [ 9549.337552] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [ 9551.791874] Key type lgssc unregistered [ 9552.049632] LNet: 337305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9552.059374] LNetError: 337305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9552.077783] LNet: Removed LNI 192.168.203.135@tcp [ 9552.807465] Key type .llcrypt unregistered [ 9552.811450] Key type ._llcrypt unregistered [ 9616.731279] Key type ._llcrypt registered [ 9616.734448] Key type .llcrypt registered [ 9616.841631] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [ 9631.467624] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [ 9632.635546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9632.669207] alg: No test for adler32 (adler32-zlib) [ 9633.742507] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9634.028974] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [ 9635.727169] Key type lgssc registered [ 9636.967058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9669.797965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9669.823687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.143523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9671.193425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9671.324434] Lustre: lustre-MDT0000: new disk, initializing [ 9671.401396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.447690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9675.360626] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9686.443345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9686.604673] Lustre: 342240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9686.728377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9686.733462] Lustre: Skipped 1 previous similar message [ 9686.805268] Lustre: lustre-MDT0001: new disk, initializing [ 9686.974939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9687.071126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9687.089764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9692.671796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9702.019458] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9707.580439] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9714.275211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9714.530652] Lustre: lustre-OST0000: new disk, initializing [ 9714.534840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9714.542862] Lustre: 344207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9714.644507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.199976] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9721.914967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9721.924168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9722.067353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9730.345257] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9735.735168] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9735.923520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9740.958922] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9741.213295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9747.946564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.964583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9747.972769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9747.986148] LustreError: Skipped 1 previous similar message [ 9747.995184] Lustre: Skipped 1 previous similar message [ 9749.396909] Lustre: server umount lustre-OST0000 complete [ 9753.568765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9753.577398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.588618] Lustre: Skipped 1 previous similar message [ 9753.594885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9758.180490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9758.188869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9759.928957] Lustre: server umount lustre-MDT0000 complete [ 9763.559286] LustreError: 344217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450703 with bad export cookie 5123757841358037530 [ 9763.563270] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9763.569722] LustreError: 344217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9763.963525] Lustre: server umount lustre-MDT0001 complete [ 9771.697597] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:51:50 (1788450710) [ 9778.382806] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:51:56 (1788450716) [ 9788.479287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.853848] LustreError: 346961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.867638] LustreError: 346961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9788.926633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.662405] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9794.034938] LustreError: 346961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9798.114925] LustreError: 346962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.236862] LustreError: 346961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9804.000295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.372935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9809.373599] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9817.063262] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9823.007937] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9829.103060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9829.431948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9836.333943] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9843.796486] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9851.642460] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9853.082682] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9854.574765] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9856.170724] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9857.798537] Lustre: Failing over lustre-MDT0000 [ 9858.042749] Lustre: server umount lustre-MDT0000 complete [ 9859.044503] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9859.051657] LustreError: 348500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.061050] Lustre: Skipped 2 previous similar messages [ 9859.079605] LustreError: 348500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9869.281535] LustreError: 346962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.299792] LustreError: 346962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 9874.400512] Lustre: 340463:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450798/real 1788450798] req@000000000ba479aa x1875326348105344/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788450814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9874.433826] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9875.128885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9884.646771] LustreError: 340460:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d6f35644 x1875326348110848/t0(0) o250->MGC192.168.203.135@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9884.957792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9885.008060] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9886.855050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9888.778371] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9890.306668] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9890.357502] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9890.424323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9897.352658] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9898.653236] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9901.644543] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9903.131838] Lustre: Failing over lustre-MDT0000 [ 9903.334599] Lustre: server umount lustre-MDT0000 complete [ 9905.633214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.635947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9905.636381] LustreError: 346958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9905.636388] LustreError: 346958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 9905.643591] Lustre: Skipped 2 previous similar messages [ 9920.919719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9920.989717] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9921.176375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9921.228843] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9923.174460] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9925.386843] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9926.645371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9926.661092] Lustre: Skipped 2 previous similar messages [ 9926.679754] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9926.718222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9933.677866] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9934.888303] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9941.991611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9941.995093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9942.002860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9942.002868] Lustre: Skipped 1 previous similar message [ 9942.012059] Lustre: Skipped 1 previous similar message [ 9946.961459] Lustre: server umount lustre-MDT0000 complete [ 9947.118203] LustreError: 347691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.145756] LustreError: 347691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 9949.960150] LustreError: 346941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450889 with bad export cookie 5123757841358040239 [ 9949.962752] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9949.975379] LustreError: 346941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9950.257996] Lustre: server umount lustre-MDT0001 complete [ 9963.654936] Lustre: server umount lustre-OST0000 complete [ 9977.505039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.919494] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [ 9993.894755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.320591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9994.327410] Lustre: Skipped 1 previous similar message [ 9998.706811] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10005.168611] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10009.958925] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10016.091122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10022.435769] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10022.653851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10028.754257] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10033.122440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.122581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10033.133541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10033.133550] Lustre: Skipped 2 previous similar messages [10033.134762] Lustre: Skipped 2 previous similar messages [10039.276317] Lustre: server umount lustre-OST0000 complete [10043.367192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10043.376630] Lustre: Skipped 4 previous similar messages [10048.780446] Lustre: server umount lustre-MDT0000 complete [10052.167667] LustreError: 353281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450992 with bad export cookie 5123757841358040981 [10052.172390] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10052.186032] LustreError: 353281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10052.472061] Lustre: server umount lustre-MDT0001 complete [10063.621380] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10066.772355] Key type lgssc unregistered [10067.066867] LNet: 356833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10067.077508] LNetError: 356833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10067.100748] LNet: Removed LNI 192.168.203.135@tcp [10067.941268] Key type .llcrypt unregistered [10067.943099] Key type ._llcrypt unregistered [10095.245909] Key type ._llcrypt registered [10095.251683] Key type .llcrypt registered [10095.320241] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10096.354543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10096.397088] alg: No test for adler32 (adler32-zlib) [10097.538226] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10097.802182] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10099.511159] Key type lgssc registered [10100.660111] Lustre: Echo OBD driver; http://www.lustre.org/ [10108.761522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10108.771605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.385748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10115.233151] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10120.685466] LustreError: 358348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.705231] LustreError: 358348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10124.418393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.718990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.450581] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10136.834684] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10142.121865] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10148.087067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10148.405332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10154.265991] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10155.576270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10160.173663] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10170.848302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10170.852575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.860670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.877269] Lustre: Skipped 2 previous similar messages [10170.887372] Lustre: Skipped 2 previous similar messages [10175.975346] LustreError: 358349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.992243] LustreError: 358349:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10176.051810] Lustre: server umount lustre-MDT0000 complete [10179.180213] LustreError: 358329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451119 with bad export cookie 10134643138741052958 [10179.184520] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.189119] LustreError: 358329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10179.481571] Lustre: server umount lustre-MDT0001 complete [10192.331387] Lustre: server umount lustre-OST0000 complete [10204.738594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10205.021553] LustreError: 362059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10205.094340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10209.282760] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10210.275544] LustreError: 362060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10214.370970] LustreError: 362059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10218.071713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10218.467994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10223.079886] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10229.670377] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.353095] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10242.642839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10243.077803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10249.220296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10249.290782] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10256.263517] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10269.678968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10269.687893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.697385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10269.701142] Lustre: Skipped 2 previous similar messages [10269.705287] Lustre: Skipped 1 previous similar message [10274.791230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10274.799964] Lustre: Skipped 3 previous similar messages [10275.198714] Lustre: server umount lustre-MDT0000 complete [10278.675169] LustreError: 362040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451218 with bad export cookie 10134643138741054071 [10278.685646] LustreError: 362040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10278.698935] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10279.106438] Lustre: server umount lustre-MDT0001 complete [10293.817933] Lustre: server umount lustre-OST0000 complete [10308.433148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.762706] LustreError: 365972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10308.841747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10313.001192] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10320.501416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.354329] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10331.844829] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10336.839348] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10342.908837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10343.178583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10343.186025] Lustre: Skipped 1 previous similar message [10348.813546] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10353.466121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10355.218120] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10361.832911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10361.842591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10361.865159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10363.873065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10363.883677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.447372] Lustre: server umount lustre-OST0000 complete [10368.993424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10368.995744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10369.002413] Lustre: Skipped 1 previous similar message [10369.033587] Lustre: Skipped 3 previous similar messages [10375.088714] Lustre: server umount lustre-MDT0000 complete [10378.660983] LustreError: 365954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451318 with bad export cookie 10134643138741055093 [10378.674512] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10378.679899] LustreError: 365954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10379.079705] Lustre: server umount lustre-MDT0001 complete [10391.036989] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10394.049782] Key type lgssc unregistered [10394.374781] LNet: 369519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10394.385561] LNetError: 369519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10394.422329] LNet: Removed LNI 192.168.203.135@tcp [10395.499931] Key type .llcrypt unregistered [10395.502873] Key type ._llcrypt unregistered [10419.506857] Key type ._llcrypt registered [10419.510245] Key type .llcrypt registered [10419.587516] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10420.918705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10420.971296] alg: No test for adler32 (adler32-zlib) [10422.105079] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10422.331252] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10424.007636] Key type lgssc registered [10424.918851] Lustre: Echo OBD driver; http://www.lustre.org/ [10432.343877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10432.373097] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.387579] LustreError: 370959:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10432.431141] LustreError: 370959:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10432.440824] LustreError: 370959:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10432.446698] LustreError: 370959:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10432.461808] LustreError: 370959:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10440.489483] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10440.501647] LustreError: 371356:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10440.527662] LustreError: 371356:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10440.538535] LustreError: 371356:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10440.545637] LustreError: 371356:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10440.554785] LustreError: 371356:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10459.089571] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10462.413277] Key type lgssc unregistered [10462.722505] LNet: 372363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10462.729875] LNetError: 372363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10462.760255] LNet: Removed LNI 192.168.203.135@tcp [10463.671622] Key type .llcrypt unregistered [10463.676906] Key type ._llcrypt unregistered [10482.371629] Key type ._llcrypt registered [10482.374839] Key type .llcrypt registered [10482.465593] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [10499.501854] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10501.395337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10501.448218] alg: No test for adler32 (adler32-zlib) [10502.502532] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10502.734763] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10504.447637] Key type lgssc registered [10505.448498] Lustre: Echo OBD driver; http://www.lustre.org/ [10543.175970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10543.194751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.535200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10544.565863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10544.660708] Lustre: lustre-MDT0000: new disk, initializing [10544.740874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10544.771629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10549.690646] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10560.779426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.949179] Lustre: 375521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10561.051251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10561.058586] Lustre: Skipped 1 previous similar message [10561.131367] Lustre: lustre-MDT0001: new disk, initializing [10561.244572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10561.300611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10561.318543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10565.785909] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10574.868723] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.275836] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10586.627906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.012449] Lustre: lustre-OST0000: new disk, initializing [10587.018494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10587.026753] Lustre: 377489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10587.126593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10592.339589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10592.347018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10592.496961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10593.293151] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10602.301609] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10607.304456] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10607.511688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10611.675915] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10611.858384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10618.342312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10618.355705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.362448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10618.378439] Lustre: Skipped 1 previous similar message [10619.790708] Lustre: server umount lustre-OST0000 complete [10623.456533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.467531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10623.483399] Lustre: Skipped 1 previous similar message [10623.498359] Lustre: Skipped 2 previous similar messages [10628.579624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10628.594294] Lustre: Skipped 1 previous similar message [10629.550768] Lustre: server umount lustre-MDT0000 complete [10633.027412] LustreError: 377488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451572 with bad export cookie 9133307569863530801 [10633.028861] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10633.038372] LustreError: 377488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10633.292661] Lustre: server umount lustre-MDT0001 complete [10641.536885] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10643.273433] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:06:21 (1788451581) [10649.857266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10650.270424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10655.014202] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10655.727304] LustreError: 379951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.759217] LustreError: 379951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10660.836478] LustreError: 379950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10663.072417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.598797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10668.105573] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10675.245927] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10681.117858] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.026831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10688.397602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10694.129452] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10700.602729] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10707.330667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10707.493465] Lustre: lustre-OST0001: new disk, initializing [10707.496848] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10707.501850] Lustre: 382787:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10707.578928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10713.007528] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10713.176346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10713.182281] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10713.300463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10721.532917] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10728.928083] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10728.931789] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.947551] LustreError: Skipped 2 previous similar messages [10728.977729] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10728.986359] Lustre: Skipped 1 previous similar message [10732.087817] Lustre: server umount lustre-OST0001 complete [10734.065634] LustreError: 383164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.084374] LustreError: 383164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10739.177425] LustreError: 381771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10739.189290] LustreError: 381771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10744.290968] LustreError: 381771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10744.304599] LustreError: 381771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10749.412810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10749.415101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10749.421966] Lustre: Skipped 2 previous similar messages [10749.424778] Lustre: Skipped 1 previous similar message [10754.479976] Lustre: server umount lustre-OST0000 complete [10759.651413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.695059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10765.212863] Lustre: server umount lustre-MDT0000 complete [10769.384485] LustreError: 379932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451709 with bad export cookie 9133307569863531683 [10769.386937] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10769.393144] LustreError: 379932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10775.720274] Lustre: server umount lustre-MDT0001 complete [10785.681417] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [10788.693427] Key type lgssc unregistered [10789.022305] LNet: 384774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10789.037187] LNetError: 384774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10789.063852] LNet: Removed LNI 192.168.203.135@tcp [10789.967139] Key type .llcrypt unregistered [10789.968808] Key type ._llcrypt unregistered [10839.133452] Key type ._llcrypt registered [10839.136861] Key type .llcrypt registered [10839.260718] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [10840.357573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10840.392194] alg: No test for adler32 (adler32-zlib) [10841.449478] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10841.708623] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [10843.455656] Key type lgssc registered [10844.757949] Lustre: Echo OBD driver; http://www.lustre.org/ [10853.397097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10853.418298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.779792] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10854.804896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10854.927915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10859.669867] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10868.025333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.115158] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10868.149056] Lustre: 387590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10868.431899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10873.197781] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10880.294321] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10886.067842] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10889.186300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10889.198711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.220123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10892.769175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10892.770690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.587272] Lustre: server umount lustre-MDT0000 complete [10898.405694] LustreError: 387582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451838 with bad export cookie 13338345788766710935 [10898.409698] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.421437] LustreError: 387582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10898.935562] Lustre: server umount lustre-MDT0001 complete [10908.858838] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:10:46 (1788451846) [10916.041828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10916.686650] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10916.720406] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10916.766940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10920.644273] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10921.955219] LustreError: 390106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.072356] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.137143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.482429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10933.854933] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10941.186702] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10945.906530] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10951.646060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10951.852985] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10952.038137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10958.915158] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [10967.146189] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10975.817925] Lustre: Permanently deactivating lustre-OST0000 [10975.832566] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10983.459418] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10985.211701] LustreError: 393091:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10985.218618] LustreError: 393091:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10985.223818] LustreError: 393091:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10988.842341] LustreError: 393140:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10988.850193] LustreError: 393140:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10997.337698] Lustre: server umount lustre-OST0000 complete [11000.133808] LustreError: 393389:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11000.143811] LustreError: 393389:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11006.432770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11006.442957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.460674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.473072] Lustre: Skipped 1 previous similar message [11012.067270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11012.074991] Lustre: Skipped 2 previous similar messages [11015.206065] Lustre: server umount lustre-MDT0000 complete [11017.185779] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.198622] LustreError: 390105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11018.616379] LustreError: 390085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451958 with bad export cookie 13338345788766711516 [11018.620252] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11018.637498] LustreError: 390085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11029.213301] Lustre: server umount lustre-MDT0001 complete [11037.440538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.085346] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11044.559363] LustreError: 394550:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11050.791710] Lustre: 394889:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.135@tcp [11052.970721] Lustre: server umount MGS complete [11063.977753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.282905] LustreError: 395612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11064.306684] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11064.311356] Lustre: Skipped 1 previous similar message [11064.357956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11069.252908] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11074.537734] LustreError: 395613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.578239] LustreError: 395613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11079.310700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.660971] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11084.992521] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11092.898308] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11098.692057] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11100.686391] Lustre: Permanently reactivating lustre-OST0000 [11100.705245] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11100.721363] Lustre: Skipped 2 previous similar messages [11129.823190] LustreError: 397441:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11140.811566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11141.162410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11141.175261] Lustre: Skipped 1 previous similar message [11148.199037] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11155.782037] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11163.263754] Lustre: server umount lustre-OST0000 complete [11169.761997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11169.765607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11169.775031] Lustre: Skipped 2 previous similar messages [11169.784833] Lustre: Skipped 1 previous similar message [11177.061295] Lustre: server umount lustre-MDT0000 complete [11180.004218] LustreError: 395608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11180.026665] LustreError: 395608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11180.424510] LustreError: 395593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452120 with bad export cookie 13338345788766712566 [11180.437243] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11180.452541] LustreError: 395593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11190.912833] Lustre: server umount lustre-MDT0001 complete [11203.036857] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11205.586501] Key type lgssc unregistered [11205.843329] LNet: 400078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11205.853855] LNetError: 400078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11205.879471] LNet: Removed LNI 192.168.203.135@tcp [11206.780248] Key type .llcrypt unregistered [11206.782159] Key type ._llcrypt unregistered [11224.970413] Key type ._llcrypt registered [11224.973417] Key type .llcrypt registered [11225.114716] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [11242.676507] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11243.252175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11243.355591] alg: No test for adler32 (adler32-zlib) [11244.463687] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11244.938690] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11246.679298] Key type lgssc registered [11247.533693] Lustre: Echo OBD driver; http://www.lustre.org/ [11282.867634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11282.880687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.112818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11284.145302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11284.206753] Lustre: lustre-MDT0000: new disk, initializing [11284.276475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11284.295743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11288.480830] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11298.156370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.303978] Lustre: 403239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11298.386666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11298.395060] Lustre: Skipped 1 previous similar message [11298.486788] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11298.496396] Lustre: lustre-MDT0001: new disk, initializing [11298.610293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11298.653379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11298.671701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11303.264678] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11312.461798] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11317.486152] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11323.304177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11323.589820] Lustre: lustre-OST0000: new disk, initializing [11323.594853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11323.608467] Lustre: 405202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11323.707378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11329.327953] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11331.130594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11331.140340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11331.233545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11337.988683] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11342.418852] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11342.703189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11347.087769] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11347.301700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11352.041636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11352.048997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11352.050317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11352.070788] Lustre: Skipped 1 previous similar message [11352.082375] Lustre: Skipped 1 previous similar message [11355.596508] Lustre: server umount lustre-OST0000 complete [11360.230696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11360.243488] LustreError: Skipped 1 previous similar message [11360.248268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11360.260159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.272503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.272983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.301018] Lustre: Skipped 1 previous similar message [11365.319736] Lustre: server umount lustre-MDT0000 complete [11368.494737] LustreError: 403231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452308 with bad export cookie 7788459260508302695 [11368.498937] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11368.510438] LustreError: 403231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11368.744801] Lustre: server umount lustre-MDT0001 complete [11375.305931] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11376.918174] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:18:35 (1788452315) [11383.803143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11384.573526] LustreError: 407663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.597523] LustreError: 407663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11384.716658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11389.921684] LustreError: 407664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.881467] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11395.041965] LustreError: 407663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.162996] LustreError: 407664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.350832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11400.730483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11405.263166] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11414.159777] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11420.437697] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11427.206466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11427.597486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11434.040640] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11442.022827] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11443.874545] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11446.356366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11455.462414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11455.469130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11455.496994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11457.890669] Lustre: server umount lustre-OST0000 complete [11461.601221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11461.621376] Lustre: Skipped 1 previous similar message [11466.722532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11466.731752] Lustre: Skipped 5 previous similar messages [11467.664464] Lustre: server umount lustre-MDT0000 complete [11471.731248] LustreError: 407645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452411 with bad export cookie 7788459260508303577 [11471.735986] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11471.741479] LustreError: 407645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11472.296454] Lustre: server umount lustre-MDT0001 complete [11484.429788] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11487.704664] Key type lgssc unregistered [11488.052063] LNet: 411392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11488.062615] LNetError: 411392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11488.087267] LNet: Removed LNI 192.168.203.135@tcp [11488.952348] Key type .llcrypt unregistered [11488.955526] Key type ._llcrypt unregistered [11515.684527] Key type ._llcrypt registered [11515.686548] Key type .llcrypt registered [11515.826609] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11516.238308] Key type .llcrypt unregistered [11516.240065] Key type ._llcrypt unregistered [11535.721545] Key type ._llcrypt registered [11535.729374] Key type .llcrypt registered [11535.829638] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11537.005738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11537.050299] alg: No test for adler32 (adler32-zlib) [11538.533256] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11538.894673] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11540.663385] Key type lgssc registered [11542.191271] Lustre: Echo OBD driver; http://www.lustre.org/ [11551.638354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11551.668884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11553.317168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11558.245771] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11563.509871] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11563.542731] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11567.986348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11568.285251] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11568.328765] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11568.338992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11573.731718] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11573.759306] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11574.948382] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11578.853128] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11578.884033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11578.896874] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11582.138108] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11583.982953] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11584.029941] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11589.325435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11589.707179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11594.367655] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11604.451599] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11604.460482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.475513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11604.965485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.977544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11604.978965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11608.204781] Lustre: server umount lustre-OST0000 complete [11615.201239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11615.212355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11615.218170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11615.227639] Lustre: Skipped 1 previous similar message [11619.075409] Lustre: server umount lustre-MDT0000 complete [11620.324834] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.342183] LustreError: 413612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11623.161745] LustreError: 415718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452563 with bad export cookie 6965009174595320819 [11623.165959] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11623.196582] LustreError: 415718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11623.535444] Lustre: server umount lustre-MDT0001 complete [11634.630901] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11637.730655] Key type lgssc unregistered [11638.119404] LNet: 416763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11638.127597] LNetError: 416763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11639.149368] LNet: Removed LNI 192.168.203.135@tcp [11640.159634] Key type .llcrypt unregistered [11640.162317] Key type ._llcrypt unregistered [11665.760668] Key type ._llcrypt registered [11665.763119] Key type .llcrypt registered [11665.899516] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11667.378134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11667.416908] alg: No test for adler32 (adler32-zlib) [11668.551229] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11668.818228] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11670.521191] Key type lgssc registered [11671.857924] Lustre: Echo OBD driver; http://www.lustre.org/ [11679.786400] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11679.817833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11695.391750] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11700.511961] LustreError: 418156:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.135@tcp: failed processing log, type 4: rc = -110 [11727.263731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11733.701908] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11740.772760] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11747.161624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11747.666657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11747.733316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11752.462507] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11752.929714] LustreError: 418182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.954063] LustreError: 418182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11758.048543] LustreError: 418181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.257354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.622266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11765.028726] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11771.946685] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11776.984216] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11783.650996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11783.666208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.686254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11785.380095] Lustre: server umount lustre-OST0000 complete [11790.821178] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11790.825238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11790.848272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11791.404714] Lustre: server umount lustre-MDT0000 complete [11792.355195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11792.357437] LustreError: 419199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.403052] LustreError: 419199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11795.006127] LustreError: 418162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452734 with bad export cookie 3369826363251285567 [11795.014456] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11795.019660] LustreError: 418162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11804.830262] Lustre: server umount lustre-MDT0001 complete [11815.686265] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11818.667917] Key type lgssc unregistered [11819.013217] LNet: 421728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11819.019073] LNetError: 421728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11819.032760] LNet: Removed LNI 192.168.203.135@tcp [11819.938212] Key type .llcrypt unregistered [11819.951153] Key type ._llcrypt unregistered [11847.515392] Key type ._llcrypt registered [11847.517578] Key type .llcrypt registered [11847.661849] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11848.746309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11848.773393] alg: No test for adler32 (adler32-zlib) [11849.828273] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11850.061340] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11851.805248] Key type lgssc registered [11853.367878] Lustre: Echo OBD driver; http://www.lustre.org/ [11861.397419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11861.440518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.941563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11866.873812] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11872.229517] LustreError: 423145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.252180] LustreError: 423145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11874.730350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11875.054279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11880.067750] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11887.347734] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11893.208492] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11900.172296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11900.570071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11907.267844] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [11907.773361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11914.874927] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11919.457641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.460820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11919.471401] Lustre: Skipped 1 previous similar message [11919.484143] Lustre: Skipped 1 previous similar message [11921.559279] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping) [11924.963321] LustreError: 423141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11925.000134] LustreError: 423141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11925.030544] Lustre: server umount lustre-MDT0000 complete [11926.658866] LustreError: 423141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.711889] LustreError: 423126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.35@tcp arrived at 1788452867 with bad export cookie 18089946485383300786 [11927.732820] LustreError: 423126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11930.083207] LustreError: 423145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.099073] LustreError: 425722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452869 with bad export cookie 18089946485383299890 [11930.111352] LustreError: 423145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11930.134581] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11930.141263] LustreError: 425722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11935.202148] LustreError: 423875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.217713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11935.238468] LustreError: 423875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11935.272284] Lustre: Skipped 1 previous similar message [11935.285353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11936.503440] Lustre: server umount lustre-OST0000 complete [11944.096106] LustreError: 423877:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000191b2f18 x1875328671712128/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11944.288387] Lustre: server umount lustre-MDT0001 complete [11954.563858] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [11957.668843] Key type lgssc unregistered [11957.983784] LNet: 426831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11957.997276] LNetError: 426831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11958.012044] LNet: Removed LNI 192.168.203.135@tcp [11958.761314] Key type .llcrypt unregistered [11958.763554] Key type ._llcrypt unregistered [11989.166650] Key type ._llcrypt registered [11989.168445] Key type .llcrypt registered [11989.218799] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [11990.509387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11990.563325] alg: No test for adler32 (adler32-zlib) [11991.713702] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11992.088876] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [11993.791216] Key type lgssc registered [11995.006769] Lustre: Echo OBD driver; http://www.lustre.org/ [12003.341868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12003.375783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12004.937652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12011.121179] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12015.073413] LustreError: 428251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.091431] LustreError: 428251:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12020.193213] LustreError: 428252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.944223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.400844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12026.332566] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12033.754724] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12039.869722] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12046.580287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12046.931283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12050.039729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12053.108243] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12060.274372] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12065.763495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12065.789816] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12065.801233] Lustre: Skipped 1 previous similar message [12068.459377] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [12070.884886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12070.888025] Lustre: Skipped 1 previous similar message [12071.220737] Lustre: server umount lustre-MDT0001 complete [12076.002990] LustreError: 430255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12076.028559] LustreError: 430255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12079.008749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12079.009730] LustreError: 428251:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.024424] Lustre: Skipped 1 previous similar message [12079.025357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12083.924520] Lustre: server umount lustre-OST0000 complete [12084.193693] LustreError: 428979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.357729] Lustre: server umount lustre-MDT0000 complete [12105.560810] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12109.404651] Key type lgssc unregistered [12109.784970] LNet: 431940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12109.796855] LNetError: 431940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12109.849344] LNet: Removed LNI 192.168.203.135@tcp [12110.988550] Key type .llcrypt unregistered [12110.991100] Key type ._llcrypt unregistered [12142.342667] Key type ._llcrypt registered [12142.348684] Key type .llcrypt registered [12142.490952] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [12162.161619] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12164.051267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12164.067509] alg: No test for adler32 (adler32-zlib) [12165.171164] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12165.609552] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [12167.343200] Key type lgssc registered [12168.769310] Lustre: Echo OBD driver; http://www.lustre.org/ [12210.144298] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12227.195071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12227.227663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.542314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12228.585480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12228.695451] Lustre: lustre-MDT0000: new disk, initializing [12228.810233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.848045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12233.946739] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12251.348680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12251.973170] Lustre: lustre-OST0000: new disk, initializing [12252.001196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12252.010944] Lustre: Skipped 1 previous similar message [12252.020836] Lustre: 436978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12252.166891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12258.961743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12258.990176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12259.031253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12260.321755] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12271.027418] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12287.077428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.204809] Lustre: 435962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12287.322978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12287.453855] Lustre: lustre-MDT0001: new disk, initializing [12287.656539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12287.697613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12287.715410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12292.208414] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12302.034682] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12302.276756] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12306.751588] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12307.031609] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12314.244459] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [12315.114650] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.130806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12315.150917] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12319.333441] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [12319.346188] Lustre: Skipped 1 previous similar message [12324.457837] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping) [12324.469482] Lustre: Skipped 2 previous similar messages [12326.367519] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12326.695736] Lustre: server umount lustre-MDT0001 complete [12335.586513] LustreError: 435967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.602878] LustreError: 435967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12340.707586] LustreError: 435972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12340.743152] LustreError: 435972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12345.826889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.829133] LustreError: 435973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12345.842855] Lustre: Skipped 1 previous similar message [12345.848256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12345.893841] Lustre: Skipped 2 previous similar messages [12350.926627] Lustre: server umount lustre-OST0000 complete [12350.954551] LustreError: 435972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12361.064684] Lustre: server umount lustre-MDT0000 complete [12375.258932] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12378.140781] Key type lgssc unregistered [12378.467233] LNet: 440728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12378.475435] LNetError: 440728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12378.506054] LNet: Removed LNI 192.168.203.135@tcp [12379.422047] Key type .llcrypt unregistered [12379.432440] Key type ._llcrypt unregistered [12424.823467] Key type ._llcrypt registered [12424.826684] Key type .llcrypt registered [12424.925507] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [12425.268630] Key type .llcrypt unregistered [12425.271614] Key type ._llcrypt unregistered [12442.062511] Key type ._llcrypt registered [12442.072126] Key type .llcrypt registered [12442.324085] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [12444.360970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12444.479405] alg: No test for adler32 (adler32-zlib) [12445.695399] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12446.123361] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [12447.911393] Key type lgssc registered [12449.525816] Lustre: Echo OBD driver; http://www.lustre.org/ [12481.248365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12481.321579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.520378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12482.552898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12482.597994] Lustre: lustre-MDT0000: new disk, initializing [12482.646967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12482.673740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12487.065491] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12497.067416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12497.259641] Lustre: 444110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12497.327666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12497.333389] Lustre: Skipped 1 previous similar message [12497.451815] Lustre: lustre-MDT0001: new disk, initializing [12497.577451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12497.635051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12497.656605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12502.494469] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12512.534977] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12517.250277] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12523.196697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12523.379376] Lustre: lustre-OST0000: new disk, initializing [12523.383741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12523.391503] Lustre: 446078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12523.456787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12529.046282] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12530.790458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12530.808425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12530.983490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12538.894257] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12546.532894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.538209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12546.548451] Lustre: Skipped 2 previous similar messages [12546.551389] Lustre: Skipped 1 previous similar message [12551.663887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12551.667823] Lustre: Skipped 3 previous similar messages [12552.250779] Lustre: server umount lustre-MDT0000 complete [12556.163077] LustreError: 446077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453496 with bad export cookie 8557468847925621321 [12556.173334] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12556.177898] LustreError: 446077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12556.608839] Lustre: server umount lustre-MDT0001 complete [12569.639729] Lustre: server umount lustre-OST0000 complete [12582.734230] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:38:40 (1788453520) [12590.591743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12591.245807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12595.301664] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12596.704901] LustreError: 448104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12596.721935] LustreError: 448104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12601.831191] LustreError: 448105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12603.881625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.389282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12609.385684] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12616.738397] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12622.382859] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12642.412857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12642.660338] Lustre: Found index 0 for lustre-OST0000, updating log [12642.901559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12648.957514] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [12656.875804] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12662.458038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12667.874957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12667.888764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12667.920175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12667.929094] Lustre: Skipped 1 previous similar message [12669.301448] Lustre: server umount lustre-OST0000 complete [12685.280266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12685.282343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12685.303612] Lustre: Skipped 2 previous similar messages [12685.326197] Lustre: Skipped 3 previous similar messages [12690.400864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12690.409630] Lustre: Skipped 1 previous similar message [12691.452251] Lustre: server umount lustre-MDT0000 complete [12694.497041] LustreError: 448100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12694.506309] LustreError: 448100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12695.015159] LustreError: 449948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453634 with bad export cookie 8557468847925622707 [12695.023042] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12695.027406] LustreError: 449948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12695.424665] Lustre: server umount lustre-MDT0001 complete [12704.974389] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:40:42 (1788453642) [12710.367292] LNet: 1 local NIs in recovery (showing 1): 192.168.203.135@tcp [13324.528904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13324.982242] LustreError: 454517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.056690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13329.688990] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13330.400030] LustreError: 454518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13330.425336] LustreError: 454518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13334.504077] LustreError: 454517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13338.852995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13339.141816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13343.530424] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13351.025865] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13357.478715] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13378.495870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13378.755390] Lustre: Found index 0 for lustre-OST0000, updating log [13379.162179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13386.097580] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13387.242168] LNet: 1 local NIs in recovery (showing 1): 192.168.203.135@tcp [13393.314750] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13396.962968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13396.972949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13396.981368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13401.597849] Lustre: server umount lustre-OST0000 complete [13405.670173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13405.678854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13405.691409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13406.760238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13407.779087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13407.785946] Lustre: Skipped 3 previous similar messages [13411.411527] LustreError: 457317:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000004a58d9b8) refcount nonzero (1) after lock cleanup; forcing cleanup. [13411.629796] Lustre: server umount lustre-MDT0000 complete [13411.888346] LustreError: 454517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13411.913093] LustreError: 454517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13415.151090] LustreError: 454497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454355 with bad export cookie 8557468847925623799 [13415.155996] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13415.165458] LustreError: 454497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13415.420812] Lustre: server umount lustre-MDT0001 complete [13442.132859] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [13451.743929] LNet: 1 local NIs in recovery (showing 1): 192.168.203.135@tcp [13452.374820] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [13488.493609] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:53:46 (1788454426) [13503.244535] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [13506.272714] Key type lgssc unregistered [13506.621814] LNet: 461129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13506.626783] LNetError: 461129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13587.582148] LNet: Removed LNI 192.168.203.135@tcp [13588.910338] Key type .llcrypt unregistered [13588.913544] Key type ._llcrypt unregistered [13605.232248] Key type ._llcrypt registered [13605.243164] Key type .llcrypt registered [13605.444161] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [13606.242803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13606.443782] alg: No test for adler32 (adler32-zlib) [13607.621937] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13607.968717] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [13609.695346] Key type lgssc registered [13610.594985] Lustre: Echo OBD driver; http://www.lustre.org/ [13614.781060] LNet: Added LNI 192.168.203.135@tcp100 [8/256/0/180] [13615.561288] LNet: Added LNI 192.168.203.135@tcp101 [8/256/0/180] [13616.330596] LNet: Added LNI 192.168.203.135@tcp102 [8/256/0/180] [13617.315299] LNet: Added LNI 192.168.203.135@tcp103 [8/256/0/180] [13618.305039] LNet: Added LNI 192.168.203.135@tcp104 [8/256/0/180] [13619.225697] LNet: Added LNI 192.168.203.135@tcp105 [8/256/0/180] [13620.176380] LNet: Added LNI 192.168.203.135@tcp106 [8/256/0/180] [13620.995150] LNet: Added LNI 192.168.203.135@tcp107 [8/256/0/180] [13621.930870] LNet: Added LNI 192.168.203.135@tcp108 [8/256/0/180] [13622.799263] LNet: Added LNI 192.168.203.135@tcp109 [8/256/0/180] [13623.682313] LNet: Added LNI 192.168.203.135@tcp110 [8/256/0/180] [13624.681368] LNet: Added LNI 192.168.203.135@tcp111 [8/256/0/180] [13625.640466] LNet: Added LNI 192.168.203.135@tcp112 [8/256/0/180] [13626.576688] LNet: Added LNI 192.168.203.135@tcp113 [8/256/0/180] [13627.596333] LNet: Added LNI 192.168.203.135@tcp114 [8/256/0/180] [13628.594934] LNet: Added LNI 192.168.203.135@tcp115 [8/256/0/180] [13629.584834] LNet: Added LNI 192.168.203.135@tcp116 [8/256/0/180] [13630.530225] LNet: Added LNI 192.168.203.135@tcp117 [8/256/0/180] [13631.437297] LNet: Added LNI 192.168.203.135@tcp118 [8/256/0/180] [13632.303979] LNet: Added LNI 192.168.203.135@tcp119 [8/256/0/180] [13633.156074] LNet: Added LNI 192.168.203.135@tcp120 [8/256/0/180] [13634.262056] LNet: Added LNI 192.168.203.135@tcp121 [8/256/0/180] [13635.209342] LNet: Added LNI 192.168.203.135@tcp122 [8/256/0/180] [13636.139985] LNet: Added LNI 192.168.203.135@tcp123 [8/256/0/180] [13637.116235] LNet: Added LNI 192.168.203.135@tcp124 [8/256/0/180] [13638.075976] LNet: Added LNI 192.168.203.135@tcp125 [8/256/0/180] [13638.985727] LNet: Added LNI 192.168.203.135@tcp126 [8/256/0/180] [13639.962335] LNet: Added LNI 192.168.203.135@tcp127 [8/256/0/180] [13640.758429] LNet: Added LNI 192.168.203.135@tcp128 [8/256/0/180] [13641.606652] LNet: Added LNI 192.168.203.135@tcp129 [8/256/0/180] [13642.530190] LNet: Added LNI 192.168.203.135@tcp130 [8/256/0/180] [13643.443743] LNet: Added LNI 192.168.203.135@tcp131 [8/256/0/180] [13644.375025] LNet: Added LNI 192.168.203.135@tcp132 [8/256/0/180] [13645.419408] LNet: Added LNI 192.168.203.135@tcp133 [8/256/0/180] [13646.233594] LNet: Added LNI 192.168.203.135@tcp134 [8/256/0/180] [13647.264412] LNet: Added LNI 192.168.203.135@tcp135 [8/256/0/180] [13654.607092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13654.627436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13655.801102] Lustre: 464159:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.135@tcp: can't add NID 192.168.203.135@tcp130: rc = -75 [13655.911518] LNetError: 464185:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13655.939670] LNetError: Protocol error connecting to 192.168.203.135@tcp113 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13655.982212] LNetError: Skipped 9 previous similar messages [13656.098226] LNet: 7 local NIs in recovery (showing 7): 192.168.203.135@tcp124, 192.168.203.135@tcp125, 192.168.203.135@tcp126, 192.168.203.135@tcp128, 192.168.203.135@tcp129, 192.168.203.135@tcp123, 192.168.203.135@tcp127 [13656.230557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13656.301717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13656.462494] Lustre: lustre-MDT0000: new disk, initializing [13656.593583] Lustre: 464215:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.135@tcp: can't add NID 192.168.203.135@tcp130: rc = -75 [13656.610559] Lustre: 464215:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13656.643532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13656.668849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13657.123697] LNetError: 464181:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13657.138430] LNetError: 464181:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [13657.155253] LNetError: Protocol error connecting to 192.168.203.135@tcp125 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13657.172407] LNetError: Skipped 10 previous similar messages [13658.145692] LNetError: 461616:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13658.186267] LNetError: Protocol error connecting to 192.168.203.135@tcp124 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13662.240106] LNetError: 464180:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13662.265272] LNetError: Protocol error connecting to 192.168.203.135@tcp124 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13662.664229] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13670.435900] LNetError: 461614:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13670.478706] LNetError: Protocol error connecting to 192.168.203.135@tcp124 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13675.643475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13675.810407] Lustre: 464176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13675.899716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13675.909586] Lustre: Skipped 1 previous similar message [13676.001732] Lustre: lustre-MDT0001: new disk, initializing [13676.036362] Lustre: 464978:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.135@tcp: can't add NID 192.168.203.135@tcp130: rc = -75 [13676.044112] Lustre: 464978:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13676.141673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13676.180286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13676.192985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13680.874460] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13684.306628] Lustre: 465366:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.135@tcp: can't add NID 192.168.203.135@tcp130: rc = -75 [13684.319025] Lustre: 465366:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13685.797341] LNetError: 464185:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13685.809206] LNetError: Protocol error connecting to 192.168.203.135@tcp119 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13690.558910] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13696.724383] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13703.876991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13704.225650] LNetError: 461615:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13704.231748] LNetError: 461615:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [13704.239610] LNetError: Protocol error connecting to 192.168.203.135@tcp128 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13704.255434] Lustre: lustre-OST0000: new disk, initializing [13704.258104] LNetError: Skipped 14 previous similar messages [13704.266483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13704.272961] Lustre: 466152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13704.419422] Lustre: 466157:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.135@tcp: can't add NID 192.168.203.135@tcp130: rc = -75 [13704.434401] Lustre: 466157:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13704.513596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13709.947712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13709.962717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13710.166236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13711.899259] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13715.488831] LNet: 1 local NIs in recovery (showing 1): 192.168.203.135@tcp124 [13721.448836] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13730.787450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13730.788093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13730.799706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13730.810406] Lustre: Skipped 1 previous similar message [13730.826609] Lustre: Skipped 1 previous similar message [13735.707473] Lustre: server umount lustre-OST0000 complete [13736.993753] LNetError: 461614:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.135 [13737.018093] LNetError: 461614:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 24 previous similar messages [13737.036410] LNetError: Protocol error connecting to 192.168.203.135@tcp119 at host 192.168.203.135:988: is it running a compatible version of Lustre? [13737.050821] LNetError: Skipped 24 previous similar messages [13741.035048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13741.068837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13741.075657] Lustre: Skipped 1 previous similar message [13745.756360] Lustre: server umount lustre-MDT0000 complete [13749.176101] LustreError: 464169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454689 with bad export cookie 4871668748349909212 [13749.177079] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13749.189893] LustreError: 464169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13749.591230] Lustre: server umount lustre-MDT0001 complete [13763.447361] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [13767.896581] Key type lgssc unregistered [13768.440210] LNet: 468069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13768.447139] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13768.455424] LNet: Removed LNI 192.168.203.135@tcp [13769.503735] LNet: Removed LNI 192.168.203.135@tcp100 [13769.527911] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13769.540285] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13769.585696] LNet: Removed LNI 192.168.203.135@tcp101 [13770.591656] LNet: Removed LNI 192.168.203.135@tcp102 [13770.594387] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13770.610112] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13770.657228] LNet: Removed LNI 192.168.203.135@tcp103 [13771.679455] LNet: Removed LNI 192.168.203.135@tcp104 [13772.703183] LNet: Removed LNI 192.168.203.135@tcp105 [13772.706327] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13772.710436] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13772.714335] LNet: Removed LNI 192.168.203.135@tcp106 [13773.727315] LNet: Removed LNI 192.168.203.135@tcp107 [13774.751134] LNet: Removed LNI 192.168.203.135@tcp108 [13775.775162] LNet: Removed LNI 192.168.203.135@tcp109 [13776.799564] LNet: Removed LNI 192.168.203.135@tcp110 [13776.808739] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13776.830084] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13776.850642] LNet: Removed LNI 192.168.203.135@tcp111 [13777.888251] LNet: Removed LNI 192.168.203.135@tcp112 [13778.975581] LNet: Removed LNI 192.168.203.135@tcp113 [13780.015519] LNet: Removed LNI 192.168.203.135@tcp114 [13781.087148] LNet: Removed LNI 192.168.203.135@tcp115 [13782.111584] LNet: Removed LNI 192.168.203.135@tcp116 [13783.140099] LNet: Removed LNI 192.168.203.135@tcp117 [13784.161959] LNet: Removed LNI 192.168.203.135@tcp118 [13785.183122] LNet: Removed LNI 192.168.203.135@tcp119 [13785.185241] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13785.188612] LNetError: 468069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13785.191727] LNet: Removed LNI 192.168.203.135@tcp120 [13786.207232] LNet: Removed LNI 192.168.203.135@tcp121 [13787.231153] LNet: Removed LNI 192.168.203.135@tcp122 [13788.255942] LNet: Removed LNI 192.168.203.135@tcp123 [13788.258905] LNet: Removed LNI 192.168.203.135@tcp124 [13789.279147] LNet: Removed LNI 192.168.203.135@tcp125 [13790.303267] LNet: Removed LNI 192.168.203.135@tcp126 [13791.332149] LNet: Removed LNI 192.168.203.135@tcp127 [13792.351297] LNet: Removed LNI 192.168.203.135@tcp128 [13793.375183] LNet: Removed LNI 192.168.203.135@tcp129 [13793.387577] LNet: Removed LNI 192.168.203.135@tcp130 [13793.406564] LNet: Removed LNI 192.168.203.135@tcp131 [13793.410402] LNet: Removed LNI 192.168.203.135@tcp132 [13793.429428] LNet: Removed LNI 192.168.203.135@tcp133 [13793.444904] LNet: Removed LNI 192.168.203.135@tcp134 [13793.463791] LNet: Removed LNI 192.168.203.135@tcp135 [13794.449292] Key type .llcrypt unregistered [13794.450968] Key type ._llcrypt unregistered [13822.060561] Key type ._llcrypt registered [13822.063487] Key type .llcrypt registered [13822.159918] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [13839.205533] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [13840.741674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13840.812557] alg: No test for adler32 (adler32-zlib) [13842.060502] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13842.432798] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [13844.180539] Key type lgssc registered [13845.244557] Lustre: Echo OBD driver; http://www.lustre.org/ [13881.428890] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:00:19 (1788454819) [13899.818303] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [13903.420408] Key type lgssc unregistered [13903.946581] LNet: 472161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13903.971434] LNetError: 472161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13903.990106] LNet: Removed LNI 192.168.203.135@tcp [13904.969515] Key type .llcrypt unregistered [13904.975767] Key type ._llcrypt unregistered [13928.053367] Key type ._llcrypt registered [13928.055409] Key type .llcrypt registered [13928.204340] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [13929.587163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13929.804211] alg: No test for adler32 (adler32-zlib) [13930.940606] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13931.161640] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [13932.865168] Key type lgssc registered [13934.330264] Lustre: Echo OBD driver; http://www.lustre.org/ [13944.626747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13944.683567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13946.204990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13946.258584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13946.391241] Lustre: lustre-MDT0000: new disk, initializing [13946.509140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13946.537692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13950.824842] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13964.270829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13964.365098] Lustre: 473387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13964.433772] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13964.439347] Lustre: Skipped 1 previous similar message [13964.556637] Lustre: lustre-MDT0001: new disk, initializing [13964.683607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13964.745794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13964.774749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13970.565666] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [13982.225807] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13989.236564] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13996.608217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13996.969295] Lustre: lustre-OST0000: new disk, initializing [13996.977895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13996.987135] Lustre: 475360:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13997.088067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14004.091188] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14004.328947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14004.341301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14004.558486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14014.632427] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14018.787351] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing lnet_if_list [14025.338169] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14025.373699] LNet: Added LNI 192.168.203.135@tcp10 [8/256/0/180] [14033.828786] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14033.877401] LNet: Added LNI 192.168.203.135@tcp11 [8/256/0/180] [14040.340304] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14040.366386] LNet: Added LNI 192.168.203.135@tcp12 [8/256/0/180] [14046.682692] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14046.731848] LNet: Added LNI 192.168.203.135@tcp13 [8/256/0/180] [14052.499712] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14052.550522] LNet: Added LNI 192.168.203.135@tcp14 [8/256/0/180] [14057.709873] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14057.774775] LNet: Added LNI 192.168.203.135@tcp15 [8/256/0/180] [14063.269460] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14063.303727] LNet: Added LNI 192.168.203.135@tcp16 [8/256/0/180] [14069.431436] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14069.469570] LNet: Added LNI 192.168.203.135@tcp17 [8/256/0/180] [14075.147212] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14075.191536] LNet: Added LNI 192.168.203.135@tcp18 [8/256/0/180] [14080.651510] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14080.689370] LNet: Added LNI 192.168.203.135@tcp19 [8/256/0/180] [14088.027952] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14088.068832] LNet: Added LNI 192.168.203.135@tcp20 [8/256/0/180] [14096.231354] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14096.252925] LNet: Added LNI 192.168.203.135@tcp21 [8/256/0/180] [14105.077265] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14105.160652] LNet: Added LNI 192.168.203.135@tcp22 [8/256/0/180] [14112.789939] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14112.861364] LNet: Added LNI 192.168.203.135@tcp23 [8/256/0/180] [14120.666865] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14120.738517] LNet: Added LNI 192.168.203.135@tcp24 [8/256/0/180] [14128.471277] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14128.511914] LNet: Added LNI 192.168.203.135@tcp25 [8/256/0/180] [14136.056079] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14136.097467] LNet: Added LNI 192.168.203.135@tcp26 [8/256/0/180] [14142.785618] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14142.826941] LNet: Added LNI 192.168.203.135@tcp27 [8/256/0/180] [14150.355970] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14150.405913] LNet: Added LNI 192.168.203.135@tcp28 [8/256/0/180] [14158.207185] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14158.263930] LNet: Added LNI 192.168.203.135@tcp29 [8/256/0/180] [14165.931677] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14165.966685] LNet: Added LNI 192.168.203.135@tcp30 [8/256/0/180] [14174.085947] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14174.114277] LNet: Added LNI 192.168.203.135@tcp31 [8/256/0/180] [14181.176464] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14181.216860] LNet: Added LNI 192.168.203.135@tcp32 [8/256/0/180] [14188.136678] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14188.175969] LNet: Added LNI 192.168.203.135@tcp33 [8/256/0/180] [14195.118216] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14195.150313] LNet: Added LNI 192.168.203.135@tcp34 [8/256/0/180] [14204.156388] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14204.216486] LNet: Added LNI 192.168.203.135@tcp35 [8/256/0/180] [14212.857624] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14212.906875] LNet: Added LNI 192.168.203.135@tcp36 [8/256/0/180] [14222.332634] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14222.389838] LNet: Added LNI 192.168.203.135@tcp37 [8/256/0/180] [14229.171615] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14229.212565] LNet: Added LNI 192.168.203.135@tcp38 [8/256/0/180] [14237.740165] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14237.777624] LNet: Added LNI 192.168.203.135@tcp39 [8/256/0/180] [14246.185483] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14246.224313] LNet: Added LNI 192.168.203.135@tcp40 [8/256/0/180] [14255.499847] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14255.541598] LNet: Added LNI 192.168.203.135@tcp41 [8/256/0/180] [14264.493295] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14264.532325] LNet: Added LNI 192.168.203.135@tcp42 [8/256/0/180] [14273.512500] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14273.537847] LNet: Added LNI 192.168.203.135@tcp43 [8/256/0/180] [14283.852796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14283.878299] LNet: Added LNI 192.168.203.135@tcp44 [8/256/0/180] [14294.187981] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14294.241664] LNet: Added LNI 192.168.203.135@tcp45 [8/256/0/180] [14303.077976] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14303.131033] LNet: Added LNI 192.168.203.135@tcp46 [8/256/0/180] [14312.438830] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14312.510599] LNet: Added LNI 192.168.203.135@tcp47 [8/256/0/180] [14322.826715] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14322.913911] LNet: Added LNI 192.168.203.135@tcp48 [8/256/0/180] [14335.382846] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14335.422791] LNet: Added LNI 192.168.203.135@tcp49 [8/256/0/180] [14345.128963] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14345.156884] LNet: Added LNI 192.168.203.135@tcp50 [8/256/0/180] [14368.226255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14368.236577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14368.251195] Lustre: Skipped 2 previous similar messages [14368.263326] Lustre: Skipped 2 previous similar messages [14371.711663] Lustre: server umount lustre-MDT0000 complete [14375.789892] LustreError: 473380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455315 with bad export cookie 10416989672907023659 [14375.800734] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14375.830221] LustreError: 473380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14376.385896] Lustre: server umount lustre-MDT0001 complete [14394.849959] Lustre: 472658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455318/real 1788455318] req@000000005dd781fd x1875330855007104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14394.915756] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14396.895681] Lustre: 472660:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455320/real 1788455320] req@00000000a07ed50c x1875330855007232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14399.165666] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [14399.974740] Lustre: 472660:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455323/real 1788455323] req@00000000287582e1 x1875330855007360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14402.015280] Lustre: 472657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455325/real 1788455325] req@0000000045fa1365 x1875330855007616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14406.111209] Lustre: 472658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455330/real 1788455330] req@000000004e0e3104 x1875330855007744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14408.749323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14421.922711] LustreError: 475353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14421.947213] LustreError: 475353:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14422.023675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14424.033180] LustreError: 475354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14424.052303] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14427.196457] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14429.156788] LustreError: 475353:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14434.291423] LustreError: 475354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14436.864510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14437.332132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14439.404577] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14442.090476] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14449.839671] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14455.376693] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14475.233901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14475.242192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14475.252924] Lustre: Skipped 1 previous similar message [14480.358668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14480.367632] Lustre: Skipped 1 previous similar message [14480.513060] Lustre: server umount lustre-OST0000 complete [14485.482239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14485.485794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14485.513377] Lustre: Skipped 2 previous similar messages [14485.533169] Lustre: Skipped 1 previous similar message [14490.602237] LustreError: 499611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14490.679767] Lustre: server umount lustre-MDT0000 complete [14495.457638] LustreError: 475359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455435 with bad export cookie 10416989672907026382 [14495.471029] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14495.906591] Lustre: server umount lustre-MDT0001 complete [14508.860151] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [14511.697916] Key type lgssc unregistered [14512.163937] LNet: 502414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14512.175950] LNetError: 502414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14513.247987] LNet: Removed LNI 192.168.203.135@tcp [14513.256315] LNetError: 502414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14513.266245] LNet: Removed LNI 192.168.203.135@tcp10 [14513.268582] LNet: Removed LNI 192.168.203.135@tcp11 [14513.270875] LNet: Removed LNI 192.168.203.135@tcp12 [14513.273200] LNet: Removed LNI 192.168.203.135@tcp13 [14513.275409] LNet: Removed LNI 192.168.203.135@tcp14 [14513.277517] LNet: Removed LNI 192.168.203.135@tcp15 [14513.279570] LNet: Removed LNI 192.168.203.135@tcp16 [14513.281734] LNet: Removed LNI 192.168.203.135@tcp17 [14513.283847] LNet: Removed LNI 192.168.203.135@tcp18 [14513.285893] LNet: Removed LNI 192.168.203.135@tcp19 [14513.287950] LNet: Removed LNI 192.168.203.135@tcp20 [14513.289934] LNet: Removed LNI 192.168.203.135@tcp21 [14513.291937] LNet: Removed LNI 192.168.203.135@tcp22 [14513.294145] LNet: Removed LNI 192.168.203.135@tcp23 [14513.296263] LNet: Removed LNI 192.168.203.135@tcp24 [14513.298278] LNet: Removed LNI 192.168.203.135@tcp25 [14513.300230] LNet: Removed LNI 192.168.203.135@tcp26 [14513.302286] LNet: Removed LNI 192.168.203.135@tcp27 [14513.304411] LNet: Removed LNI 192.168.203.135@tcp28 [14513.306497] LNet: Removed LNI 192.168.203.135@tcp29 [14513.308557] LNet: Removed LNI 192.168.203.135@tcp30 [14513.310547] LNet: Removed LNI 192.168.203.135@tcp31 [14513.312332] LNet: Removed LNI 192.168.203.135@tcp32 [14513.314498] LNet: Removed LNI 192.168.203.135@tcp33 [14513.316693] LNet: Removed LNI 192.168.203.135@tcp34 [14513.318752] LNet: Removed LNI 192.168.203.135@tcp35 [14513.320727] LNet: Removed LNI 192.168.203.135@tcp36 [14513.322987] LNet: Removed LNI 192.168.203.135@tcp37 [14513.325191] LNet: Removed LNI 192.168.203.135@tcp38 [14513.327299] LNet: Removed LNI 192.168.203.135@tcp39 [14513.329293] LNet: Removed LNI 192.168.203.135@tcp40 [14513.344151] LNet: Removed LNI 192.168.203.135@tcp41 [14513.346386] LNet: Removed LNI 192.168.203.135@tcp42 [14513.348687] LNet: Removed LNI 192.168.203.135@tcp43 [14513.351473] LNet: Removed LNI 192.168.203.135@tcp44 [14513.353698] LNet: Removed LNI 192.168.203.135@tcp45 [14513.357247] LNet: Removed LNI 192.168.203.135@tcp46 [14513.360594] LNet: Removed LNI 192.168.203.135@tcp47 [14513.365051] LNet: Removed LNI 192.168.203.135@tcp48 [14513.367569] LNet: Removed LNI 192.168.203.135@tcp49 [14513.378865] LNet: Removed LNI 192.168.203.135@tcp50 [14514.078872] Key type .llcrypt unregistered [14514.082832] Key type ._llcrypt unregistered [14539.477879] Key type ._llcrypt registered [14539.491981] Key type .llcrypt registered [14539.708225] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [14557.575267] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [14558.507661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14558.629351] alg: No test for adler32 (adler32-zlib) [14559.821901] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14560.228840] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [14561.991117] Key type lgssc registered [14563.383981] Lustre: Echo OBD driver; http://www.lustre.org/ [14599.580967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14599.602981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14600.953889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14601.028748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14601.178653] Lustre: lustre-MDT0000: new disk, initializing [14601.356336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14601.383307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14606.165865] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14617.362912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14617.608538] Lustre: 505615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14617.710518] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14617.721626] Lustre: Skipped 1 previous similar message [14617.816294] Lustre: lustre-MDT0001: new disk, initializing [14617.907522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14617.966193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14617.988734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14622.338642] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14632.042256] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14639.346619] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14650.316292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14650.859878] Lustre: *** cfs_fail_loc=90f, val=0*** [14650.977767] Lustre: lustre-OST0000: new disk, initializing [14650.981540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14650.991610] Lustre: 507633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14651.140515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14656.595484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14656.608700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14656.705941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14657.823768] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14666.322573] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14677.477838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14677.497214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14677.508398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14677.538736] Lustre: Skipped 1 previous similar message [14682.602574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14682.610648] Lustre: Skipped 2 previous similar messages [14687.200257] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14687.404433] Lustre: server umount lustre-OST0000 complete [14692.858430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14692.878655] Lustre: Skipped 1 previous similar message [14692.892562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14697.940765] Lustre: server umount lustre-MDT0000 complete [14701.786903] LustreError: 505605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455641 with bad export cookie 4848911690936278624 [14701.788233] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14701.804298] LustreError: 505605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14702.378679] Lustre: server umount lustre-MDT0001 complete [14715.098516] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [14717.982734] Key type lgssc unregistered [14718.253622] LNet: 509544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14718.259909] LNetError: 509544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14719.333971] LNet: Removed LNI 192.168.203.135@tcp [14720.264619] Key type .llcrypt unregistered [14720.266260] Key type ._llcrypt unregistered [14740.856413] Key type ._llcrypt registered [14740.862175] Key type .llcrypt registered [14741.016942] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [14741.953453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14742.008314] alg: No test for adler32 (adler32-zlib) [14743.294520] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14743.546442] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [14745.272265] Key type lgssc registered [14746.830533] Lustre: Echo OBD driver; http://www.lustre.org/ [14754.491210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14754.501603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14755.993845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14760.715465] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14765.541938] LustreError: 510818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14765.560217] LustreError: 510818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14769.119069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14769.558751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14773.374515] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14781.070544] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14787.173553] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14796.086760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14796.643512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14804.076930] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14811.187636] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14825.426609] loop0: detected capacity change from 0 to 204800000 [14825.605451] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14825.614814] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14825.622547] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14831.075108] loop0: detected capacity change from 0 to 204800000 [14831.353541] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14831.360494] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14831.368480] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14836.381434] loop0: detected capacity change from 0 to 204800000 [14842.149532] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14842.432939] Lustre: fs15246-OST0000: new disk, initializing [14842.438724] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14842.444401] Lustre: 514412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14842.540314] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14849.436743] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14861.177919] loop1: detected capacity change from 0 to 204800000 [14867.699848] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14867.855387] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14867.935053] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14868.045076] Lustre: fs15246-MDT0000: new disk, initializing [14868.317405] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14868.405292] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14868.691101] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14869.793917] Lustre: 515532:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d62c98cf x1875331706305920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788455825 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14874.224144] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [14878.900995] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14878.912320] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14878.969366] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14901.752083] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14901.759982] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14904.152935] Lustre: server umount fs15246-MDT0000 complete [14906.856699] LustreError: 510813:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14911.977471] LustreError: 512819:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14913.894543] Lustre: server umount fs15246-OST0000 complete [14928.864645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14928.872841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14928.879344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14932.455241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14932.460674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14932.483294] Lustre: Skipped 1 previous similar message [14937.569813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14937.572654] Lustre: Skipped 1 previous similar message [14942.175136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14942.307250] Lustre: server umount lustre-OST0000 complete [14947.812498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14947.824701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14947.848299] Lustre: Skipped 1 previous similar message [14947.863795] Lustre: Skipped 1 previous similar message [14952.828269] Lustre: server umount lustre-MDT0000 complete [14952.931258] LustreError: 510817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14952.945123] LustreError: 510817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14956.831920] LustreError: 510798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455896 with bad export cookie 8910356768895706110 [14956.835778] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14956.846253] LustreError: 510798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14957.531093] Lustre: server umount lustre-MDT0001 complete [14969.110000] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [14972.233291] Key type lgssc unregistered [14972.499943] LNet: 518636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14972.504353] LNetError: 518636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14972.520173] LNet: Removed LNI 192.168.203.135@tcp [14973.520645] Key type .llcrypt unregistered [14973.522549] Key type ._llcrypt unregistered [14993.522384] Key type ._llcrypt registered [14993.527656] Key type .llcrypt registered [14993.649987] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [14994.530500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14994.551903] alg: No test for adler32 (adler32-zlib) [14995.692775] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14995.909871] LNet: Added LNI 192.168.203.135@tcp [8/256/0/180] [14997.551893] Key type lgssc registered [14998.378337] Lustre: Echo OBD driver; http://www.lustre.org/ [15040.736852] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [15051.622467] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [15086.768739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15086.785144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15088.174921] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15088.200128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15088.354833] Lustre: lustre-MDT0000: new disk, initializing [15088.418264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15088.459198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15093.466731] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [15103.571482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15103.689817] Lustre: 523106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15103.753949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15103.757944] Lustre: Skipped 1 previous similar message [15103.828660] Lustre: lustre-MDT0001: new disk, initializing [15103.957292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15104.012627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15104.031577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15107.803461] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [15118.186172] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15123.633702] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15131.601606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15131.839423] Lustre: lustre-OST0000: new disk, initializing [15131.843520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15131.851594] Lustre: 525074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15131.947287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15137.963796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug -1 all [15139.421698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15139.448042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15139.621311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15148.665844] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15155.919236] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15156.501515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15163.337953] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15163.648753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15170.542153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15170.545632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15170.560093] Lustre: Skipped 1 previous similar message [15170.576239] Lustre: Skipped 1 previous similar message [15172.485345] Lustre: server umount lustre-OST0000 complete [15180.770257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15180.775851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15180.800941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15180.813433] Lustre: Skipped 1 previous similar message [15183.256774] Lustre: server umount lustre-MDT0000 complete [15185.889550] LustreError: 523117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15185.911745] LustreError: 523117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15187.680114] LustreError: 523099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456127 with bad export cookie 8129048382293320772 [15187.680921] LustreError: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15187.688065] LustreError: 523099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15188.218163] Lustre: server umount lustre-MDT0001 complete [15213.969392] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_hostid [15224.867943] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing load_modules_local [15258.643266] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14987 sec ========= 13:23:16 (1788456196) [15260.379733] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:23:18 (1788456198) === [15264.401335] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:23:22 (1788456202) === [15309.053604] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing unload_modules_local [15312.003254] Key type lgssc unregistered [15312.370024] LNet: 531723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15312.379369] LNetError: 531723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15312.412643] LNet: Removed LNI 192.168.203.135@tcp [15313.645979] Key type .llcrypt unregistered [15313.650432] Key type ._llcrypt unregistered