[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491076347 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003418] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.007994] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011151] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022035] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027433] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028188] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030159] *** VALIDATE tmpfs *** [ 0.032165] *** VALIDATE proc *** [ 0.033282] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035311] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038045] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042570] debug: unmapping init [mem 0xffffffff8f659000-0xffffffff8f660fff] [ 0.044200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045813] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046032] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051022] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.054265] rcu: Hierarchical SRCU implementation. [ 0.056473] smp: Bringing up secondary CPUs ... [ 0.057573] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.061355] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.216731] node 0 deferred pages initialised in 149ms [ 0.219037] devtmpfs: initialized [ 0.220481] x86/mm: Memory block size: 128MB [ 0.225211] gcov: version magic: 0x41383552 [ 0.226741] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229260] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.232416] pinctrl core: initialized pinctrl subsystem [ 0.234220] [ 0.234815] ************************************************************* [ 0.237021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.239019] ** ** [ 0.242017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244017] ** ** [ 0.246017] ** This means that this kernel is built to expose internal ** [ 0.249019] ** IOMMU data structures, which may compromise security on ** [ 0.251018] ** your system. ** [ 0.254017] ** ** [ 0.255017] ** If you see this message and you are not debugging the ** [ 0.257015] ** kernel, report this immediately to your vendor! ** [ 0.259021] ** ** [ 0.260017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262011] ************************************************************* [ 0.264346] NET: Registered protocol family 16 [ 0.266545] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.271061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.274185] cpuidle: using governor menu [ 0.275782] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.276450] PCI: Using configuration type 1 for base access [ 0.278132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.288050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293048] cryptd: max_cpu_qlen set to 1000 [ 0.296255] ACPI: Added _OSI(Module Device) [ 0.298020] ACPI: Added _OSI(Processor Device) [ 0.300015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.301017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306230] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.313527] ACPI: Interpreter enabled [ 0.314070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.316015] ACPI: Using IOAPIC for interrupt routing [ 0.318133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.321533] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.331765] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.334056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.336036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.340094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.345536] acpiphp: Slot [2] registered [ 0.346170] acpiphp: Slot [5] registered [ 0.348151] acpiphp: Slot [6] registered [ 0.350136] acpiphp: Slot [7] registered [ 0.351142] acpiphp: Slot [8] registered [ 0.353153] acpiphp: Slot [9] registered [ 0.354150] acpiphp: Slot [10] registered [ 0.356134] acpiphp: Slot [3] registered [ 0.357148] acpiphp: Slot [4] registered [ 0.358142] acpiphp: Slot [11] registered [ 0.360119] acpiphp: Slot [12] registered [ 0.362098] acpiphp: Slot [13] registered [ 0.363130] acpiphp: Slot [14] registered [ 0.365127] acpiphp: Slot [15] registered [ 0.366120] acpiphp: Slot [16] registered [ 0.368146] acpiphp: Slot [17] registered [ 0.369211] acpiphp: Slot [18] registered [ 0.371114] acpiphp: Slot [19] registered [ 0.373142] acpiphp: Slot [20] registered [ 0.374113] acpiphp: Slot [21] registered [ 0.376128] acpiphp: Slot [22] registered [ 0.377172] acpiphp: Slot [23] registered [ 0.379180] acpiphp: Slot [24] registered [ 0.381127] acpiphp: Slot [25] registered [ 0.382173] acpiphp: Slot [26] registered [ 0.383119] acpiphp: Slot [27] registered [ 0.385149] acpiphp: Slot [28] registered [ 0.386094] acpiphp: Slot [29] registered [ 0.388117] acpiphp: Slot [30] registered [ 0.389134] acpiphp: Slot [31] registered [ 0.391089] PCI host bridge to bus 0000:00 [ 0.392021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.394030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.397026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.399026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.402046] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.405021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.406192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.410145] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.413419] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.422968] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.428014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.430035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.432024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.434034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436474] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439638] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.442123] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.445970] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.450022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.464021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.469019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.475859] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.489021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.498023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.516021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.521924] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.537021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.543020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.557023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.566041] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.573020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.580025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.596025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606020] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.620026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.634023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.642900] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.647019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.652021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.662018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.669854] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.675020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.697033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.710198] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.712470] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.715496] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.717395] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.720292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.724199] iommu: Default domain type: Passthrough [ 0.726509] SCSI subsystem initialized [ 0.728162] ACPI: bus type USB registered [ 0.729119] usbcore: registered new interface driver usbfs [ 0.731105] usbcore: registered new interface driver hub [ 0.733109] usbcore: registered new device driver usb [ 0.735205] pps_core: LinuxPPS API ver. 1 registered [ 0.736012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740096] PTP clock support registered [ 0.743051] EDAC MC: Ver: 3.0.0 [ 0.745200] PCI: Using ACPI for IRQ routing [ 0.748062] NetLabel: Initializing [ 0.749014] NetLabel: domain hash size = 128 [ 0.750014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.752097] NetLabel: unlabeled traffic allowed by default [ 0.753372] vgaarb: loaded [ 0.755328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.757022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765000] clocksource: Switched to clocksource kvm-clock [ 0.875518] VFS: Disk quotas dquot_6.6.0 [ 0.876927] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.880432] *** VALIDATE ramfs *** [ 0.881610] *** VALIDATE hugetlbfs *** [ 0.883280] pnp: PnP ACPI init [ 0.885694] pnp: PnP ACPI: found 6 devices [ 0.904898] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.908759] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.910947] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.913286] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915778] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918286] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.921371] NET: Registered protocol family 2 [ 0.923842] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931211] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.935543] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.940202] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.945841] TCP: Hash tables configured (established 65536 bind 65536) [ 0.948940] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.952942] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.956601] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960199] NET: Registered protocol family 1 [ 0.963373] RPC: Registered named UNIX socket transport module. [ 0.965565] RPC: Registered udp transport module. [ 0.967389] RPC: Registered tcp transport module. [ 0.968972] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.970730] NET: Registered protocol family 44 [ 0.972223] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.973878] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.975972] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978199] PCI: CLS 0 bytes, default 64 [ 0.979872] Unpacking initramfs... [ 2.436195] debug: unmapping init [mem 0xffff99aa7cc54000-0xffff99aa7ffbffff] [ 2.440295] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.442549] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.445710] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.967901] Initialise system trusted keyrings [ 2.969661] Key type blacklist registered [ 2.974153] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.982866] zbud: loaded [ 2.985388] *** VALIDATE nfs *** [ 2.986863] *** VALIDATE nfs4 *** [ 2.988618] pstore: using deflate compression [ 2.991176] Platform Keyring initialized [ 3.099934] NET: Registered protocol family 38 [ 3.101672] Key type asymmetric registered [ 3.102913] Asymmetric key parser 'x509' registered [ 3.105177] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.108271] io scheduler mq-deadline registered [ 3.109859] io scheduler kyber registered [ 3.111674] io scheduler bfq registered [ 3.113932] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.116851] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.119574] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.123165] ACPI: Power Button [PWRF] [ 3.129644] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.137173] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.155565] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.165041] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.183599] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.212623] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.239599] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.244585] Non-volatile memory driver v1.3 [ 3.245951] Linux agpgart interface v0.103 [ 3.282181] virtio_blk virtio1: [vda] 150672 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.284360] vda: detected capacity change from 0 to 77144064 [ 3.298281] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.301528] vdb: detected capacity change from 0 to 1073741824 [ 3.315491] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.318133] vdc: detected capacity change from 0 to 2621440000 [ 3.330630] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.332816] vdd: detected capacity change from 0 to 2621440000 [ 3.345435] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.348535] vde: detected capacity change from 0 to 4294967296 [ 3.364017] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.366408] vdf: detected capacity change from 0 to 4294967296 [ 3.372906] libphy: Fixed MDIO Bus: probed [ 3.385660] usbcore: registered new interface driver usbserial_generic [ 3.387655] usbserial: USB Serial support registered for generic [ 3.389653] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.395483] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.397149] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.399333] mousedev: PS/2 mouse device common for all mice [ 3.401935] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.402329] rtc_cmos 00:05: RTC can wake from S4 [ 3.411589] rtc_cmos 00:05: registered as rtc0 [ 3.411790] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.413320] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.416873] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.418127] intel_pstate: CPU model not supported [ 3.422807] hid: raw HID events driver (C) Jiri Kosina [ 3.425108] usbcore: registered new interface driver usbhid [ 3.426782] usbhid: USB HID core driver [ 3.427952] drop_monitor: Initializing network drop monitor service [ 3.429570] Initializing XFRM netlink socket [ 3.431272] NET: Registered protocol family 10 [ 3.433846] Segment Routing with IPv6 [ 3.434872] NET: Registered protocol family 17 [ 3.436456] mpls_gso: MPLS GSO support [ 3.441820] RAS: Correctable Errors collector initialized. [ 3.444427] AVX version of gcm_enc/dec engaged. [ 3.446300] AES CTR mode by8 optimization enabled [ 3.512535] sched_clock: Marking stable (3512509684, 0)->(4458983445, -946473761) [ 3.514943] registered taskstats version 1 [ 3.516267] Loading compiled-in X.509 certificates [ 3.517693] zswap: loaded using pool lzo/zbud [ 3.546591] Key type big_key registered [ 3.559465] Key type encrypted registered [ 3.561028] ima: No TPM chip found, activating TPM-bypass! [ 3.563238] ima: Allocated hash algorithm: sha1 [ 3.564927] ima: No architecture policies found [ 3.566796] evm: Initialising EVM extended attributes: [ 3.568677] evm: security.selinux [ 3.570043] evm: security.ima [ 3.571230] evm: security.capability [ 3.572546] evm: HMAC attrs: 0x1 [ 3.574871] rtc_cmos 00:05: setting system clock to 2026-09-30 12:35:34 UTC (1790771734) [ 3.580615] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 3.583878] debug: unmapping init [mem 0xffffffff8f382000-0xffffffff8f658fff] [ 3.592081] Write protecting the kernel read-only data: 28672k [ 3.595117] debug: unmapping init [mem 0xffffffff8da03000-0xffffffff8dbfffff] [ 3.597344] debug: unmapping init [mem 0xffffffff8e314000-0xffffffff8e3fffff] [ 3.638528] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.645311] systemd[1]: Detected virtualization kvm. [ 3.647131] systemd[1]: Detected architecture x86-64. [ 3.648524] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.671824] systemd[1]: No hostname configured. [ 3.673225] systemd[1]: Set hostname to . [ 3.674970] random: systemd: uninitialized urandom read (16 bytes read) [ 3.677639] systemd[1]: Initializing machine ID from random generator. [ 3.726904] random: ln: uninitialized urandom read (6 bytes read) [ 3.821813] random: systemd: uninitialized urandom read (16 bytes read) [ 3.824119] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.828383] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.832404] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.506777] device-mapper: uevent: version 1.0.3 [ 4.509162] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.221659] random: fast init done [ 5.265500] virtio_net virtio0 ens2: renamed from eth0 [ 5.271295] scsi host0: ata_piix [ 5.312935] scsi host1: ata_piix [ 5.314705] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.317044] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.615330] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.079407] random: crng init done [ 10.080927] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.439572] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.542821] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.761519] SELinux: Disabled at runtime. [ 11.822505] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.831468] systemd[1]: Detected virtualization kvm. [ 11.833397] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.226903] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.229285] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.233445] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.237249] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.239481] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.244821] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.254112] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ 12.336430] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.623942] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.904765] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.954545] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.043341] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.056836] EDAC sbridge: Ver: 1.1.2 [ 14.376927] Key type dns_resolver registered [ 14.686880] NFS: Registering the id_resolver key type [ 14.688958] Key type id_resolver registered [ 14.690601] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg230-server login: [ 35.183451] hrtimer: interrupt took 6559909 ns [ 62.463826] libcfs: loading out-of-tree module taints kernel. [ 62.569359] Key type ._llcrypt registered [ 62.575885] Key type .llcrypt registered [ 62.666124] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 86.394637] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 88.973579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.012040] alg: No test for adler32 (adler32-zlib) [ 90.479820] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 91.529753] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 93.423310] Key type lgssc registered [ 95.316917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.157207] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.049513] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 171.569527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 171.610368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.880580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.919526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.996750] Lustre: lustre-MDT0000: new disk, initializing [ 173.060577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.075397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.705186] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.123357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.213521] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 191.243726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 191.248351] Lustre: Skipped 1 previous similar message [ 191.349475] Lustre: lustre-MDT0001: new disk, initializing [ 191.447983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 191.500805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 191.506709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.588032] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.631761] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.869487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.136612] Lustre: lustre-OST0000: new disk, initializing [ 211.139526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.143701] Lustre: 8450:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.226635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.115787] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.609531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.627629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.222077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.432315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.549579] Lustre: lustre-OST0001: new disk, initializing [ 231.552847] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.565380] Lustre: 9523:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.626412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 238.193935] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.658151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 239.661855] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 239.698962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.760725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.091967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 264.304565] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.757807] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing yml_node [ 275.617798] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 278.698442] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 281.611375] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 283.759064] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 30 08:40:12 EDT 2026 [ 306.291538] Lustre: DEBUG MARKER: only running tests: 33-75 [ 308.254448] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 310.899783] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.696347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 329.715357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.739226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.744314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.747411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.597191] Lustre: server umount lustre-MDT0000 complete [ 337.488669] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790772068 with bad export cookie 6418918932837679242 [ 337.499405] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 337.500983] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 341.984240] LustreError: 12201:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.984463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.000591] LustreError: 12201:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 342.006768] Lustre: Skipped 2 previous similar messages [ 342.007684] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.007690] Lustre: Skipped 3 previous similar messages [ 343.012042] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.032560] LustreError: 6517:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 343.837992] Lustre: server umount lustre-MDT0001 complete [ 347.563278] Lustre: server umount lustre-OST0000 complete [ 351.082912] Lustre: server umount lustre-OST0001 complete [ 357.833168] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 370.099474] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 406.687738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.061189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 407.103698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 407.209625] Lustre: lustre-MDT0000: new disk, initializing [ 407.331754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.350246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.016963] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 422.719321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.859254] Lustre: 15125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 422.933398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 422.936964] Lustre: Skipped 1 previous similar message [ 423.073838] Lustre: lustre-MDT0001: new disk, initializing [ 423.224507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.288874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.299055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 427.698556] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 437.929780] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 444.483413] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 452.577277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 452.974612] Lustre: lustre-OST0000: new disk, initializing [ 452.978236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 452.986971] Lustre: 17095:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 453.087190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.875854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 454.889437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 455.016980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 459.877738] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 471.893258] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 478.157851] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 478.380172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.502184] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 482.653298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.874160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.876480] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.879881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.879888] Lustre: Skipped 3 previous similar messages [ 485.893295] Lustre: Skipped 2 previous similar messages [ 485.924409] LustreError: Skipped 1 previous similar message [ 490.987985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.997298] Lustre: Skipped 2 previous similar messages [ 499.167980] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 499.303299] Lustre: server umount lustre-OST0000 complete [ 505.314904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.320521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.340410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 505.347273] Lustre: Skipped 2 previous similar messages [ 511.446602] Lustre: server umount lustre-MDT0000 complete [ 511.472434] LustreError: 16736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.508392] LustreError: 16736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 515.944246] LustreError: 17094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790772246 with bad export cookie 6418918932837680768 [ 515.949196] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.967979] LustreError: 17094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 516.495952] Lustre: server umount lustre-MDT0001 complete [ 521.329895] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:44:10 (1790772250) [ 524.381417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 526.457760] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:44:15 (1790772255) [ 533.347752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.776821] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.846852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.389727] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 539.110396] LustreError: 19505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.324735] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.350077] LustreError: 19504:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 549.171366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.031295] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 564.071635] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 571.346549] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 579.899818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.684422] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 595.985640] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 616.422280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 616.443486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.464917] Lustre: Skipped 1 previous similar message [ 616.473187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 616.475893] Lustre: Skipped 2 previous similar messages [ 622.123111] Lustre: server umount lustre-OST0000 complete [ 626.657471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.749946] Lustre: server umount lustre-MDT0000 complete [ 632.806192] LustreError: 20686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.827886] LustreError: 20686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 636.774676] LustreError: 19485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790772367 with bad export cookie 6418918932837681657 [ 636.780879] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.784399] LustreError: 19485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 637.211207] Lustre: server umount lustre-MDT0001 complete [ 649.728613] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 652.517971] Key type lgssc unregistered [ 652.901774] LNet: 23062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.909731] LNetError: 23062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.926908] LNet: Removed LNI 192.168.202.130@tcp [ 654.163441] Key type .llcrypt unregistered [ 654.165338] Key type ._llcrypt unregistered [ 677.936460] Key type ._llcrypt registered [ 677.940630] Key type .llcrypt registered [ 678.039881] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 679.175044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 679.319425] alg: No test for adler32 (adler32-zlib) [ 680.378277] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 680.638883] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 682.303985] Key type lgssc registered [ 683.593549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 691.873258] loop: module loaded [ 691.975538] loop0: detected capacity change from 0 to 204800000 [ 692.139100] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.155368] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.470353] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.917593] loop0: detected capacity change from 0 to 204800000 [ 699.098532] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.113473] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.130419] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.443705] loop0: detected capacity change from 0 to 204800000 [ 709.749906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 709.781413] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.081570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 711.129275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 711.223138] Lustre: lustre-MDT0000: new disk, initializing [ 711.330701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.348134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 715.512970] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 724.310991] loop1: detected capacity change from 0 to 204800000 [ 730.217990] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 730.572385] Lustre: lustre-OST07c6: new disk, initializing [ 730.578292] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 730.592525] Lustre: Skipped 1 previous similar message [ 730.599930] Lustre: 26281:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 730.679217] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 736.380396] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 738.395827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 738.411635] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 738.452147] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 741.764212] Lustre: 25268:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 741.767919] Lustre: 25268:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 741.770985] Lustre: 25268:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 741.774304] Lustre: 25268:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 741.777481] Lustre: 25268:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 741.780351] Lustre: 25268:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 755.346792] Lustre: 25268:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 755.352826] Lustre: 25268:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 755.357871] Lustre: 25268:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.364234] Lustre: 25268:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 755.371128] Lustre: 25268:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.376478] Lustre: 25268:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.620734] Lustre: 25270:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 760.644756] Lustre: 25270:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 760.667556] Lustre: 25270:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.690980] Lustre: 25270:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.717695] Lustre: 25270:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.738552] Lustre: 25270:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.764407] Lustre: 25270:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 760.771668] Lustre: 25270:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.788941] Lustre: 25270:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.804112] Lustre: 25270:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.822247] Lustre: 25270:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.846021] Lustre: 25270:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.124060] Lustre: 25268:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 764.129661] Lustre: 25268:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 764.134270] Lustre: 25268:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 764.138850] Lustre: 25268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.146357] Lustre: 25268:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.150195] Lustre: 25268:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.162588] Lustre: 25268:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 764.167853] Lustre: 25268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.172862] Lustre: 25268:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 764.176699] Lustre: 25268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 764.183892] Lustre: 25268:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 764.189589] Lustre: 25268:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.185842] Lustre: 26963:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 769.195269] Lustre: 26963:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.218602] Lustre: 26963:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.229535] Lustre: 26963:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.242244] Lustre: 26963:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.253466] Lustre: 26963:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.268625] Lustre: 26963:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 769.280084] Lustre: 26963:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.290187] Lustre: 26963:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 769.295144] Lustre: 26963:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.304274] Lustre: 26963:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.318384] Lustre: 26963:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.516771] Lustre: 26963:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 777.525642] Lustre: 26963:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages [ 777.529756] Lustre: 26963:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 777.533441] Lustre: 26963:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.537994] Lustre: 26963:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.541321] Lustre: 26963:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.546912] Lustre: 26963:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 777.552584] Lustre: 26963:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.555866] Lustre: 26963:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.561240] Lustre: 26963:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 777.566958] Lustre: 26963:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.571905] Lustre: 26963:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 830.436915] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.444052] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.449412] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 835.520328] Lustre: server umount lustre-OST07c6 complete [ 844.270681] Lustre: server umount lustre-MDT0000 complete [ 862.486488] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 865.025224] Key type lgssc unregistered [ 865.321064] LNet: 28837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.324145] LNetError: 28837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.337401] LNet: Removed LNI 192.168.202.130@tcp [ 866.144361] Key type .llcrypt unregistered [ 866.148661] Key type ._llcrypt unregistered [ 889.877712] Key type ._llcrypt registered [ 889.880337] Key type .llcrypt registered [ 890.030244] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 891.088679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.233589] alg: No test for adler32 (adler32-zlib) [ 892.339411] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 892.645388] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 894.360967] Key type lgssc registered [ 895.824828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 904.413957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 904.430618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.960852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.115516] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 915.430520] LustreError: 30143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.445799] LustreError: 30143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 919.340903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.766217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 924.697610] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 933.607285] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 940.161881] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 947.282881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 947.725289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.838326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 953.781144] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 962.561079] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 979.425703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.428911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.452322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.468573] Lustre: Skipped 1 previous similar message [ 979.486280] Lustre: Skipped 1 previous similar message [ 981.768853] Lustre: server umount lustre-OST0000 complete [ 986.603320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 986.610516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.626136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.665052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.666295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.570690] Lustre: server umount lustre-MDT0000 complete [ 994.792564] LustreError: 31627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 994.813359] LustreError: 31627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 996.290266] LustreError: 30124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790772727 with bad export cookie 1036446157597070262 [ 996.297825] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.298810] LustreError: 30124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 996.662884] Lustre: server umount lustre-MDT0001 complete [ 1008.910707] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1011.798600] Key type lgssc unregistered [ 1012.097996] LNet: 33879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.109286] LNetError: 33879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.131742] LNet: Removed LNI 192.168.202.130@tcp [ 1013.062758] Key type .llcrypt unregistered [ 1013.065498] Key type ._llcrypt unregistered [ 1033.293959] Key type ._llcrypt registered [ 1033.295758] Key type .llcrypt registered [ 1033.414172] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1034.402403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.503700] alg: No test for adler32 (adler32-zlib) [ 1035.551557] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1035.769195] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1037.424482] Key type lgssc registered [ 1038.533128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1047.392696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1047.417597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.899740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.164509] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1058.279048] LustreError: 35165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.301346] LustreError: 35165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1062.706407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.115976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.842132] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1078.344789] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1085.156811] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1093.719110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.351716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.982454] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1112.595770] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1118.916784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1128.415909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.422901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.448179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.442703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.443799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1129.444721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.565019] LustreError: 36988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.586096] LustreError: 36988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1135.015326] Lustre: server umount lustre-OST0000 complete [ 1144.806464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.821731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1144.846639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.868891] Lustre: Skipped 2 previous similar messages [ 1149.145966] Lustre: server umount lustre-MDT0000 complete [ 1149.926435] LustreError: 35165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.953716] LustreError: 35165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1155.042691] LustreError: 36739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.060932] LustreError: 36739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1155.967363] LustreError: 35146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790772886 with bad export cookie 15042410251415937608 [ 1155.995749] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.944341] Lustre: server umount lustre-MDT0001 complete [ 1172.073568] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1176.581544] Key type lgssc unregistered [ 1176.987194] LNet: 38724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.996686] LNetError: 38724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.013657] LNet: Removed LNI 192.168.202.130@tcp [ 1178.108976] Key type .llcrypt unregistered [ 1178.112156] Key type ._llcrypt unregistered [ 1205.952691] Key type ._llcrypt registered [ 1205.955791] Key type .llcrypt registered [ 1206.093379] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1207.176719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1207.263355] alg: No test for adler32 (adler32-zlib) [ 1208.461929] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1208.776906] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1210.463692] Key type lgssc registered [ 1211.594620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.444800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1220.504478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.176946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.364765] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1232.353974] LustreError: 40157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.373701] LustreError: 40157:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1236.770104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.310894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1242.443593] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1250.789744] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1256.347471] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1263.693907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1264.042573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.142540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1270.247518] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1279.111936] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.375569] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 1292.775082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.779305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.831087] Lustre: Skipped 2 previous similar messages [ 1292.855017] Lustre: Skipped 2 previous similar messages [ 1295.681862] Lustre: server umount lustre-MDT0000 complete [ 1296.410469] LustreError: 40153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.908909] LustreError: 40158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.945428] LustreError: 40158:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1300.978458] LustreError: 40137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773031 with bad export cookie 6102541574165673926 [ 1300.983445] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.995992] LustreError: 40137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1301.550256] LustreError: 40154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1301.587629] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 1303.009348] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.056903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1305.063173] Lustre: Skipped 1 previous similar message [ 1306.669067] LustreError: 40154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.683167] LustreError: 40154:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1310.178727] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1310.190948] Lustre: Skipped 1 previous similar message [ 1315.296340] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.301158] LustreError: 40153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.334901] LustreError: 40153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1315.652758] Lustre: server umount lustre-MDT0001 complete [ 1330.462506] Lustre: server umount lustre-OST0000 complete [ 1349.689128] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1353.402400] Key type lgssc unregistered [ 1353.782060] LNet: 44020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.796168] LNetError: 44020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.816935] LNet: Removed LNI 192.168.202.130@tcp [ 1354.920932] Key type .llcrypt unregistered [ 1354.926176] Key type ._llcrypt unregistered [ 1381.826396] Key type ._llcrypt registered [ 1381.845333] Key type .llcrypt registered [ 1381.975755] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1383.278231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1383.349256] alg: No test for adler32 (adler32-zlib) [ 1384.656288] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1385.040116] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1386.767220] Key type lgssc registered [ 1388.161812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1396.036378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1396.065053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.592853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.071921] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1407.974516] LustreError: 45470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.989416] LustreError: 45470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1412.685317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.965623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1417.891554] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1425.437332] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.061471] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1438.111475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.443922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.557543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1444.625588] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1451.600824] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1463.272486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1463.275167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.279132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.282576] LustreError: Skipped 1 previous similar message [ 1463.301472] Lustre: Skipped 1 previous similar message [ 1464.891031] Lustre: lustre-OST0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 1464.904337] Lustre: Skipped 1 previous similar message [ 1468.403609] LustreError: 47293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.825683] Lustre: server umount lustre-OST0000 complete [ 1488.867847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.885639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.887973] Lustre: Skipped 1 previous similar message [ 1493.985822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.995481] Lustre: Skipped 1 previous similar message [ 1494.405164] Lustre: server umount lustre-MDT0000 complete [ 1499.145470] LustreError: 45470:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.174732] LustreError: 45470:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1499.427055] LustreError: 45451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773230 with bad export cookie 17502286673960153594 [ 1499.429260] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.451671] LustreError: 45451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1499.471411] LustreError: 46200:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 1499.471475] LustreError: 46200:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 1505.978962] Lustre: server umount lustre-MDT0001 complete [ 1518.858749] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1521.536225] Key type lgssc unregistered [ 1521.829684] LNet: 49180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.835906] LNetError: 49180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.852823] LNet: Removed LNI 192.168.202.130@tcp [ 1522.706936] Key type .llcrypt unregistered [ 1522.714890] Key type ._llcrypt unregistered [ 1550.889493] Key type ._llcrypt registered [ 1550.892406] Key type .llcrypt registered [ 1551.055540] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1552.294633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.444878] alg: No test for adler32 (adler32-zlib) [ 1553.480219] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1553.743568] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1555.527733] Key type lgssc registered [ 1557.086522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1564.404928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1564.436837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.859441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.294821] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1575.407630] LustreError: 50627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.426644] LustreError: 50627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1579.042893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.479283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1583.623333] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1591.449918] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1597.974428] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1605.018418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.379118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.476311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1611.970252] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1619.951466] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1630.236727] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1633.741670] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1645.925329] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-30 9h02m54s [ 1647.218352] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1650.674260] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1650.676812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.694411] Lustre: Skipped 2 previous similar messages [ 1650.713428] Lustre: Skipped 2 previous similar messages [ 1651.750347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.832723] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [ 1653.848928] Lustre: Skipped 2 previous similar messages [ 1655.598231] Lustre: server umount lustre-MDT0000 complete [ 1657.835836] LustreError: 50627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.862264] LustreError: 50627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1659.231368] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1660.964037] LustreError: 50623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.998675] LustreError: 51355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1664.031633] LustreError: 51355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1666.005065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1666.166535] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1666.406661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.473322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1667.443557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1671.068910] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1674.429606] Lustre: DEBUG MARKER: Wait for df (36943) ... [ 1676.080336] Lustre: DEBUG MARKER: done [ 1681.376364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.382415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.395789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.398670] Lustre: Skipped 1 previous similar message [ 1681.893088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.112295] Lustre: server umount lustre-OST0000 complete [ 1693.159459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1693.163866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.188564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.210450] Lustre: Skipped 2 previous similar messages [ 1693.227209] Lustre: Skipped 4 previous similar messages [ 1697.914274] Lustre: server umount lustre-MDT0000 complete [ 1700.846776] LustreError: 50627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.875687] LustreError: 50627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1702.706821] LustreError: 53256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773433 with bad export cookie 6474120108732095728 [ 1702.711256] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1702.714906] LustreError: 53256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1703.667139] Lustre: server umount lustre-MDT0001 complete [ 1717.113893] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1720.434922] Key type lgssc unregistered [ 1720.756780] LNet: 55548:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.759411] LNetError: 55548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.829662] LNet: Removed LNI 192.168.202.130@tcp [ 1722.708855] Key type .llcrypt unregistered [ 1722.711377] Key type ._llcrypt unregistered [ 1764.626222] Key type ._llcrypt registered [ 1764.631174] Key type .llcrypt registered [ 1764.760261] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1766.141606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1766.172797] alg: No test for adler32 (adler32-zlib) [ 1767.264825] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1767.575167] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1769.215213] Key type lgssc registered [ 1770.402835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1777.863913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1777.890657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.177322] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1779.205648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1779.401178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.047162] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1791.798254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.859975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1791.879365] Lustre: 57771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1792.057635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1796.613811] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1804.655530] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.897447] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1818.056797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1818.231816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1818.487432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.409297] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 1827.537568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1832.717368] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1839.923802] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-30 9h06m09s [ 1841.337129] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1847.748800] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1850.639619] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1850.690214] LustreError: 57780:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1870.920378] Lustre: lustre-MDT0000: Client f120d8e5-b6fc-491c-b539-434c8777a585 (at 192.168.202.30@tcp) reconnecting [ 1890.711172] LustreError: 57780:0:(mdt_handler.c:2345:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1893.694107] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1895.943992] Lustre: DEBUG MARKER: done [ 1904.609697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.617717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.635631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.657043] Lustre: Skipped 1 previous similar message [ 1909.050464] Lustre: server umount lustre-OST0000 complete [ 1914.851934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.853498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1914.859982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.870657] Lustre: Skipped 2 previous similar messages [ 1915.937670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.941679] Lustre: Skipped 2 previous similar messages [ 1919.314737] Lustre: server umount lustre-MDT0000 complete [ 1923.201040] LustreError: 57765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773654 with bad export cookie 10337644351931053276 [ 1923.218250] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1923.449852] Lustre: server umount lustre-MDT0001 complete [ 1936.240563] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 1939.355592] Key type lgssc unregistered [ 1939.703475] LNet: 62099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.712597] LNetError: 62099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.730707] LNet: Removed LNI 192.168.202.130@tcp [ 1940.911324] Key type .llcrypt unregistered [ 1940.915750] Key type ._llcrypt unregistered [ 1991.073630] Key type ._llcrypt registered [ 1991.076103] Key type .llcrypt registered [ 1991.179843] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 1992.651305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1992.674773] alg: No test for adler32 (adler32-zlib) [ 1993.815472] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1994.056342] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 1995.755089] Key type lgssc registered [ 1997.057156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.100936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2001.133785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.477358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2002.491731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2002.692188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.895621] Lustre: Failing over lustre-MDT0000 [ 2004.279620] Lustre: server umount lustre-MDT0000 complete [ 2012.811462] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:09:01 (1790773741) [ 2021.696314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.364649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.303973] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2036.970529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.057087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2037.089313] Lustre: 64915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2037.323526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.636157] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2050.827691] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2058.048180] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2066.140668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2066.352116] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2066.580131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2073.652987] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2074.316384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2082.093054] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2089.181805] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2098.657933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.671451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.711732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.192137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.194528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.211067] Lustre: Skipped 1 previous similar message [ 2100.237148] Lustre: Skipped 2 previous similar messages [ 2103.701535] Lustre: server umount lustre-MDT0000 complete [ 2108.175769] LustreError: 66771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773839 with bad export cookie 17599008232222367945 [ 2108.198818] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2108.558410] Lustre: server umount lustre-MDT0001 complete [ 2113.245847] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2118.176376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2126.817142] Lustre: 63826:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773841/real 1790773841] req@00000000ec20c841 x1877762348460032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790773857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.855646] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.762949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.695501] Lustre: 63827:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773844/real 1790773844] req@00000000668b86f3 x1877762348460160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790773860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2131.935159] Lustre: 63828:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773846/real 1790773846] req@00000000ffe38f58 x1877762348460288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790773862 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2134.347331] LustreError: 66765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.376370] LustreError: 66765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2134.466682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.539666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2135.528928] LustreError: 66766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.529584] Lustre: 63828:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773849/real 1790773849] req@000000009a92be32 x1877762348460544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790773865 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.612472] LustreError: 68689:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.624840] LustreError: 68689:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2137.689252] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2140.639504] Lustre: 63826:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773854/real 1790773854] req@0000000047e4460d x1877762348460672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790773870 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2141.667294] LustreError: 66765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.700583] LustreError: 66765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2146.391251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.808954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2151.909489] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2152.216784] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2156.070104] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2161.078136] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2167.010857] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.484952] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.490664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.502268] Lustre: Skipped 2 previous similar messages [ 2176.529290] Lustre: Skipped 2 previous similar messages [ 2181.575457] Lustre: server umount lustre-MDT0000 complete [ 2181.610186] LustreError: 69421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.633664] LustreError: 69421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2185.473582] LustreError: 64906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773916 with bad export cookie 17599008232222370227 [ 2185.481920] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.491885] LustreError: 64906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2185.743572] Lustre: server umount lustre-MDT0001 complete [ 2193.016540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2202.085827] Lustre: 63826:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790773917/real 1790773917] req@00000000f65fd42e x1877762348488576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790773933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2202.108123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2202.398082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.335866] LustreError: 63825:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000984b0527 x1877762348489600/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.682301] LustreError: 66765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.771761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.814203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2214.944705] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2222.416888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.553654] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2228.197484] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2231.267388] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2235.525190] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2241.278969] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2251.750443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.762676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.767896] Lustre: Skipped 1 previous similar message [ 2251.783092] Lustre: Skipped 2 previous similar messages [ 2256.044127] Lustre: server umount lustre-MDT0000 complete [ 2256.865065] LustreError: 71373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.882033] LustreError: 71373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2259.790131] LustreError: 66771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790773990 with bad export cookie 17599008232222371606 [ 2259.791089] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.799236] LustreError: 66771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2260.015162] Lustre: server umount lustre-MDT0001 complete [ 2266.584518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2270.239634] Lustre: DEBUG MARKER: files compared the same [ 2282.640975] Lustre: server umount lustre-OST0000 complete [ 2300.425829] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2302.165292] Key type lgssc unregistered [ 2302.338483] LNet: 74647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.341029] LNetError: 74647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.352561] LNet: Removed LNI 192.168.202.130@tcp [ 2302.881456] Key type .llcrypt unregistered [ 2302.883724] Key type ._llcrypt unregistered [ 2328.289701] Key type ._llcrypt registered [ 2328.293451] Key type .llcrypt registered [ 2328.407768] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2329.338705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2329.575515] alg: No test for adler32 (adler32-zlib) [ 2330.601971] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 2330.902928] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2332.591608] Key type lgssc registered [ 2333.696964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2342.738564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2342.756749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.154399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.060265] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug +malloc all [ 2354.659626] LustreError: 76080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.680304] LustreError: 76080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2358.506101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.839316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.498693] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug +malloc all [ 2371.058158] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2377.551164] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.807712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2386.061442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.214689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2392.618279] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug +malloc all [ 2400.886246] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2414.051924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.058581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.100383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.487666] Lustre: server umount lustre-OST0000 complete [ 2424.289803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.332151] Lustre: Skipped 1 previous similar message [ 2424.356982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.370961] Lustre: Skipped 2 previous similar messages [ 2427.039412] Lustre: server umount lustre-MDT0000 complete [ 2429.410187] LustreError: 76077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.439420] LustreError: 76077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2432.146883] LustreError: 76061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774163 with bad export cookie 16314976077957026832 [ 2432.149231] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.153118] LustreError: 76061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2432.548957] Lustre: server umount lustre-MDT0001 complete [ 2446.995330] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2450.780618] Key type lgssc unregistered [ 2451.231869] LNet: 79612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.239909] LNetError: 79612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.263394] LNet: Removed LNI 192.168.202.130@tcp [ 2452.378632] Key type .llcrypt unregistered [ 2452.383555] Key type ._llcrypt unregistered [ 2480.466390] Key type ._llcrypt registered [ 2480.470593] Key type .llcrypt registered [ 2480.595761] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2481.857317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2481.900544] alg: No test for adler32 (adler32-zlib) [ 2482.911772] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 2483.146789] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2484.831200] Key type lgssc registered [ 2485.893975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2493.135969] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2493.159772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2508.703524] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2513.826642] LustreError: 81039:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [ 2540.511395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.011374] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2556.245917] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2565.118534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.282182] Lustre: *** cfs_fail_loc=706, val=0*** [ 2565.643709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.717987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2570.076960] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2570.726419] LustreError: 81064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.736334] LustreError: 81064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2575.848526] LustreError: 81065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.871751] LustreError: 81065:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2578.186897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.591403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2582.809728] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2591.005485] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2597.730321] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2601.445498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2601.452607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.477668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2604.003615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.014644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.038498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2609.642278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2609.649518] Lustre: Skipped 2 previous similar messages [ 2614.239148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2614.435194] Lustre: server umount lustre-OST0000 complete [ 2619.364941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2619.370147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.384201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.499431] Lustre: server umount lustre-MDT0000 complete [ 2624.992675] LustreError: 82129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.025656] LustreError: 82129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2628.576587] LustreError: 81045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774359 with bad export cookie 1387457371679298297 [ 2628.581191] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.590948] LustreError: 81045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2628.982600] Lustre: server umount lustre-MDT0001 complete [ 2641.927989] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 2644.988309] Key type lgssc unregistered [ 2645.288184] LNet: 84656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.299476] LNetError: 84656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.376548] LNet: Removed LNI 192.168.202.130@tcp [ 2647.469257] Key type .llcrypt unregistered [ 2647.478854] Key type ._llcrypt unregistered [ 2678.362591] Key type ._llcrypt registered [ 2678.364668] Key type .llcrypt registered [ 2678.452831] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2679.609611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2679.789112] alg: No test for adler32 (adler32-zlib) [ 2681.106268] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 2681.496792] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 2683.207258] Key type lgssc registered [ 2684.816779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2692.639730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2692.688880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.852761] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2706.877023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.262486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.599385] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2712.551373] LustreError: 86777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.579594] LustreError: 86777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2717.666980] LustreError: 86781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.691454] LustreError: 86781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2719.768028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.226475] LustreError: 86777:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.248736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.253652] LustreError: 86777:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2725.346151] LustreError: 86780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.362377] LustreError: 86780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2726.131113] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2730.472913] LustreError: 87524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.490446] LustreError: 87524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2734.722968] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.746426] Lustre: MGS: Client 50fdd6b9-54ca-4501-8f36-38fb3a93df62 (at 0@lo) reconnecting [ 2734.767389] LustreError: 86791:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2735.039329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.118619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2739.173611] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2755.569886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.570967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.577057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.586356] Lustre: Skipped 1 previous similar message [ 2759.869092] Lustre: server umount lustre-OST0000 complete [ 2765.794708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2765.827667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.834569] Lustre: Skipped 2 previous similar messages [ 2770.916622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.928081] LustreError: 88320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.941494] LustreError: 88320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2770.957763] Lustre: server umount lustre-MDT0000 complete [ 2775.140308] Lustre: server umount lustre-MDT0001 complete [ 2778.699742] Lustre: server umount MGS complete [ 2791.656146] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:22:00 (1790774520) [ 2822.744658] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 2835.080624] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 2877.298051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.160909] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2893.559537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.898718] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2893.961300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2894.094877] Lustre: lustre-MDT0001: new disk, initializing [ 2894.207880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.246261] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2894.255534] Lustre: 93969:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2894.275332] LustreError: 93969:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2899.997208] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2912.097108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2912.439891] Lustre: lustre-OST0000: new disk, initializing [ 2912.445874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2912.469395] Lustre: 94734:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2912.557472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.998532] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2920.031810] Lustre: 95024:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000053df3795 x1877763069879424/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790774666 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2930.338174] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2937.137452] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.149283] Lustre: MGS: Client 26764160-4685-472a-86f3-c24e6fec9b21 (at 0@lo) reconnecting [ 2937.185969] Lustre: 93235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2937.213232] LustreError: 93235:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2937.221460] LustreError: 93235:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2937.249362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2937.328141] Lustre: lustre-MDT0000: new disk, initializing [ 2937.454822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.472376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2937.481575] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.528787] Lustre: 95803:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000805af263 x1877763069886208/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790774685 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2941.877353] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 2943.649845] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2947.566668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2947.579757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2947.708498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2972.641419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.655850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.676335] Lustre: Skipped 2 previous similar messages [ 2972.689800] Lustre: Skipped 1 previous similar message [ 2977.763696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.775267] Lustre: Skipped 1 previous similar message [ 2978.509559] Lustre: server umount lustre-OST0000 complete [ 2982.881567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.897737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.002575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2988.009547] Lustre: Skipped 3 previous similar messages [ 2988.668929] Lustre: server umount lustre-MDT0000 complete [ 2998.915906] Lustre: server umount lustre-MDT0001 complete [ 3003.326951] Lustre: server umount MGS complete [ 3014.904342] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:25:43 (1790774743) [ 3037.759452] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3041.322523] Key type lgssc unregistered [ 3041.819760] LNet: 98305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3041.841648] LNetError: 98305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3042.865801] LNet: Removed LNI 192.168.202.130@tcp [ 3044.159135] Key type .llcrypt unregistered [ 3044.161981] Key type ._llcrypt unregistered [ 3063.933384] Key type ._llcrypt registered [ 3063.943146] Key type .llcrypt registered [ 3064.075632] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3065.047376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3065.121254] alg: No test for adler32 (adler32-zlib) [ 3066.180852] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 3066.473489] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3068.143260] Key type lgssc registered [ 3069.311431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3077.419346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3077.447451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.005564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.799740] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3088.362747] LustreError: 99547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.383510] LustreError: 99547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3092.595980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.080745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3097.697631] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3106.372677] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3113.305177] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3120.141036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.426307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.522254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3127.198160] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3135.599445] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3148.256702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3148.266123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.270949] Lustre: Skipped 1 previous similar message [ 3148.271905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3152.634372] Lustre: server umount lustre-OST0000 complete [ 3158.508347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3158.513732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3158.528023] Lustre: Skipped 1 previous similar message [ 3158.552394] Lustre: Skipped 2 previous similar messages [ 3162.826432] Lustre: server umount lustre-MDT0000 complete [ 3163.617473] LustreError: 99542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.637852] LustreError: 99542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3166.972833] LustreError: 101390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774897 with bad export cookie 16600537406819143108 [ 3166.980146] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3166.987345] LustreError: 101390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3167.328472] Lustre: server umount lustre-MDT0001 complete [ 3180.888534] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3183.796520] Key type lgssc unregistered [ 3184.093883] LNet: 103079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3184.107977] LNetError: 103079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3184.142848] LNet: Removed LNI 192.168.202.130@tcp [ 3185.198033] Key type .llcrypt unregistered [ 3185.203454] Key type ._llcrypt unregistered [ 3204.738649] Key type ._llcrypt registered [ 3204.743333] Key type .llcrypt registered [ 3204.915514] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3206.212367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3206.302399] alg: No test for adler32 (adler32-zlib) [ 3207.436600] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 3207.691084] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3209.344868] Key type lgssc registered [ 3210.707463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3216.663500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3216.679505] LustreError: 104189:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3216.695906] LustreError: 104196:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3216.701276] LustreError: 104189:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3216.711044] LustreError: 104189:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3216.723709] LustreError: 104189:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3216.728742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.244627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.654571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.833200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3230.956965] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3236.788817] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing lsmod [ 3239.849472] LustreError: 105402:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3239.859248] LustreError: 105401:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3239.868664] LustreError: 105402:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3239.879886] LustreError: 105402:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3239.890597] LustreError: 105402:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3239.902498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.472474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.588445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3247.585922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3247.597920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3247.622370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.611934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.616516] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.660247] Lustre: Skipped 1 previous similar message [ 3252.769906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.792923] Lustre: Skipped 4 previous similar messages [ 3257.835568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.847505] Lustre: Skipped 2 previous similar messages [ 3258.335186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3258.529491] Lustre: server umount lustre-MDT0000 complete [ 3262.945645] LustreError: 104230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.976654] LustreError: 104230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3263.114987] LustreError: 104212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790774994 with bad export cookie 773370370233637128 [ 3263.141656] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3265.119346] LustreError: 104664:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb604721 x1877763621411840/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3265.539508] Lustre: server umount lustre-MDT0001 complete [ 3280.333329] Lustre: server umount lustre-OST0000 complete [ 3291.855500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.376671] LustreError: 106471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.441107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.530820] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3297.762688] LustreError: 106472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3302.882644] LustreError: 106471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.821114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3306.220025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3311.275186] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3320.154531] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3326.875867] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3334.177408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3334.588826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.663945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3342.121691] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3352.086336] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3362.280910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.297684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3362.309817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.623641] Lustre: server umount lustre-OST0000 complete [ 3372.518144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.541076] Lustre: Skipped 2 previous similar messages [ 3372.541504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3372.547144] Lustre: Skipped 1 previous similar message [ 3374.809018] Lustre: server umount lustre-MDT0000 complete [ 3377.636879] LustreError: 107713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.650545] LustreError: 107713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3378.089323] LustreError: 108317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775109 with bad export cookie 773370370233638059 [ 3378.096226] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3378.097512] LustreError: 108317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3378.497479] Lustre: server umount lustre-MDT0001 complete [ 3390.129890] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3394.094809] Key type lgssc unregistered [ 3394.568068] LNet: 110004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3394.581479] LNetError: 110004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3394.634231] LNet: Removed LNI 192.168.202.130@tcp [ 3395.796952] Key type .llcrypt unregistered [ 3395.799602] Key type ._llcrypt unregistered [ 3423.827541] Key type ._llcrypt registered [ 3423.835124] Key type .llcrypt registered [ 3424.020844] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3425.343968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3425.364073] alg: No test for adler32 (adler32-zlib) [ 3426.527793] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 3426.842965] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3428.519240] Key type lgssc registered [ 3429.832327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3438.526908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3438.549235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.109248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3446.039708] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3449.319667] LustreError: 111442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.338340] LustreError: 111442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3454.432206] LustreError: 111441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3456.131252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.710703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3462.542541] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3474.303166] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3481.181390] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3490.175223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.578994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.339679] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3507.325140] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3510.072921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3517.977398] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3522.928860] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3527.656572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3527.662264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.682069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.726395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.732876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3530.760839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.832784] Lustre: server umount lustre-OST0000 complete [ 3538.912065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3538.915335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.946054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.948554] Lustre: Skipped 1 previous similar message [ 3540.960991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.961378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3540.981055] Lustre: Skipped 1 previous similar message [ 3544.728951] Lustre: server umount lustre-MDT0000 complete [ 3546.082120] LustreError: 111442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3546.089559] LustreError: 111442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3548.649532] LustreError: 111422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775279 with bad export cookie 5963246005491874043 [ 3548.667704] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3548.668255] LustreError: 111422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3549.036478] Lustre: server umount lustre-MDT0001 complete [ 3560.410566] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3562.681877] Key type lgssc unregistered [ 3562.968889] LNet: 115084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3562.981595] LNetError: 115084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3562.996306] LNet: Removed LNI 192.168.202.130@tcp [ 3564.059321] Key type .llcrypt unregistered [ 3564.061031] Key type ._llcrypt unregistered [ 3584.330231] Key type ._llcrypt registered [ 3584.339237] Key type .llcrypt registered [ 3584.524658] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3585.817031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3585.833832] alg: No test for adler32 (adler32-zlib) [ 3586.858368] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 3587.082823] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3588.751360] Key type lgssc registered [ 3589.994411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3599.005767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3599.028978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.508896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.685777] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3610.097534] LustreError: 116321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.124414] LustreError: 116321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3614.747340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.986827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.099687] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3627.615535] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3634.829681] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.806965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.225647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.328254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3650.744550] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3659.581262] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3669.986077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.989986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.996215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3670.010247] Lustre: Skipped 1 previous similar message [ 3675.107549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.115508] Lustre: Skipped 1 previous similar message [ 3675.659490] Lustre: server umount lustre-OST0000 complete [ 3680.225508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.243203] Lustre: Skipped 1 previous similar message [ 3680.250870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.263732] Lustre: Skipped 1 previous similar message [ 3685.355114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.593478] Lustre: server umount lustre-MDT0000 complete [ 3689.501565] LustreError: 116302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775420 with bad export cookie 11427584358574162699 [ 3689.519657] LustreError: 116302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3689.522186] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3689.897673] Lustre: server umount lustre-MDT0001 complete [ 3703.192110] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3706.022400] Key type lgssc unregistered [ 3706.403624] LNet: 119852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3706.412641] LNetError: 119852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3706.431673] LNet: Removed LNI 192.168.202.130@tcp [ 3707.243491] Key type .llcrypt unregistered [ 3707.245915] Key type ._llcrypt unregistered [ 3740.121217] Key type ._llcrypt registered [ 3740.123397] Key type .llcrypt registered [ 3740.271507] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3741.895501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3741.940548] alg: No test for adler32 (adler32-zlib) [ 3743.257533] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 3743.610607] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3745.399799] Key type lgssc registered [ 3747.185663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3758.227467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3758.370603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3760.221825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3765.985757] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3770.340211] LustreError: 121290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.363629] LustreError: 121290:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3775.482499] LustreError: 121291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.087968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.559574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3784.712083] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3793.848285] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3799.835488] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3806.716963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3807.040277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.476419] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3822.281975] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3830.601738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3841.399783] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3845.828052] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3845.831454] Lustre: Skipped 2 previous similar messages [ 3847.847502] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3847.896241] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3851.551092] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3851.562151] Lustre: Skipped 2 previous similar messages [ 3859.067066] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3859.071229] Lustre: Skipped 1 previous similar message [ 3865.840167] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3865.853788] Lustre: Skipped 2 previous similar messages [ 3871.383203] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.30@tcp 192.168.0.[2,10]@tcp [ 3876.035203] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.30/24@tcp in log lustre-MDT0000 [ 3876.042122] Lustre: Skipped 2 previous similar messages [ 3877.207737] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.30/24@tcp [ 3877.218390] Lustre: Skipped 1 previous similar message [ 3882.985559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3882.995377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.997332] LustreError: Skipped 1 previous similar message [ 3883.032200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.048211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.059342] LustreError: Skipped 4 previous similar messages [ 3886.065033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.081431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.083720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3887.107363] Lustre: Skipped 1 previous similar message [ 3892.194536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3892.209529] Lustre: Skipped 1 previous similar message [ 3897.311184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3897.328372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.336431] Lustre: Skipped 1 previous similar message [ 3897.440497] Lustre: server umount lustre-OST0000 complete [ 3902.432243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3902.433724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.461701] Lustre: Skipped 1 previous similar message [ 3907.553569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.557778] Lustre: Skipped 3 previous similar messages [ 3908.047843] Lustre: server umount lustre-MDT0000 complete [ 3912.631711] LustreError: 121271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775643 with bad export cookie 9163352223725360448 [ 3912.640652] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3912.644820] LustreError: 121271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3912.990180] Lustre: server umount lustre-MDT0001 complete [ 3928.002457] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 3931.049155] Key type lgssc unregistered [ 3931.564966] LNet: 126123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3931.570926] LNetError: 126123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3931.584928] LNet: Removed LNI 192.168.202.130@tcp [ 3932.470785] Key type .llcrypt unregistered [ 3932.473362] Key type ._llcrypt unregistered [ 3963.984053] Key type ._llcrypt registered [ 3963.988564] Key type .llcrypt registered [ 3964.071874] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 3964.803450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3964.867304] alg: No test for adler32 (adler32-zlib) [ 3965.913363] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 3966.091401] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 3967.759235] Key type lgssc registered [ 3969.346922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3978.306297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3978.341831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.805702] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3979.845654] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.30/24@tcp [ 3979.895145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.965391] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 3989.475126] LustreError: 127705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3989.501383] LustreError: 127705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3993.097311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.538568] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3993.572779] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.30/24@tcp [ 3993.651642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3998.374283] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4005.871323] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.314517] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4019.385876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.731374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4024.890815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4027.238282] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4036.114373] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4048.355655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4048.357197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.367172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4048.392594] Lustre: Skipped 1 previous similar message [ 4053.476032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.484033] Lustre: Skipped 2 previous similar messages [ 4053.942515] Lustre: server umount lustre-OST0000 complete [ 4058.593514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.597287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4058.620401] Lustre: Skipped 1 previous similar message [ 4058.629677] Lustre: Skipped 1 previous similar message [ 4063.713297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4064.594682] Lustre: server umount lustre-MDT0000 complete [ 4068.835434] LustreError: 127706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.873053] LustreError: 127706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4069.339685] LustreError: 127686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790775800 with bad export cookie 14302830413167501431 [ 4069.353736] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.805991] Lustre: server umount lustre-MDT0001 complete [ 4082.234579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4084.940056] Key type lgssc unregistered [ 4085.371697] LNet: 131292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4085.379330] LNetError: 131292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4085.395786] LNet: Removed LNI 192.168.202.130@tcp [ 4086.564837] Key type .llcrypt unregistered [ 4086.567122] Key type ._llcrypt unregistered [ 4113.664286] Key type ._llcrypt registered [ 4113.670683] Key type .llcrypt registered [ 4113.753119] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4131.058354] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4132.347198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4132.388183] alg: No test for adler32 (adler32-zlib) [ 4133.482217] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 4133.709896] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4135.440417] Key type lgssc registered [ 4136.720362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4174.675457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4174.712421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.008251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4176.031164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4176.108456] Lustre: lustre-MDT0000: new disk, initializing [ 4176.197783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.214357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4181.709834] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4192.990261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4193.120708] Lustre: 134711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4193.164617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4193.167745] Lustre: Skipped 1 previous similar message [ 4193.223456] Lustre: lustre-MDT0001: new disk, initializing [ 4193.305673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4193.347693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4193.361316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4198.015773] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4209.012328] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4215.685903] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4224.667960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.011565] Lustre: lustre-OST0000: new disk, initializing [ 4225.027509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4225.034873] Lustre: 136679:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4225.124273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.398262] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4233.867022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4233.883050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4233.969714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4241.363752] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4245.613920] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4245.809858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4250.580572] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4250.806867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4254.698822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4254.713583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4254.716892] LustreError: Skipped 1 previous similar message [ 4254.718730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4254.761149] Lustre: Skipped 1 previous similar message [ 4259.813138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.830957] Lustre: Skipped 2 previous similar messages [ 4264.929758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.935547] Lustre: Skipped 1 previous similar message [ 4266.975253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4267.210609] Lustre: server umount lustre-OST0000 complete [ 4275.169407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4275.172902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4275.198589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4275.206876] Lustre: Skipped 1 previous similar message [ 4275.217247] Lustre: Skipped 1 previous similar message [ 4278.003380] Lustre: server umount lustre-MDT0000 complete [ 4282.716277] LustreError: 135516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776013 with bad export cookie 3504535904112398184 [ 4282.741920] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4283.014475] Lustre: server umount lustre-MDT0001 complete [ 4293.082446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.689315] LustreError: 138899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.706965] LustreError: 138899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4293.786556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4299.228383] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4299.232726] LustreError: 138900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.338366] LustreError: 138899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.457626] LustreError: 138900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.491634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4308.865174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4314.261643] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4322.404597] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4328.867993] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.784649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4338.440621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4346.821677] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4354.659984] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4359.774302] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4360.048177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4364.697843] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4364.893844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4374.998247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.159586] Lustre: lustre-OST0001: new disk, initializing [ 4375.163188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4375.169238] Lustre: 142270:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4375.281964] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4382.457274] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4382.899937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4382.908217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4382.984214] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4391.167863] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4391.436880] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.274596] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4396.519958] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4403.080280] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d538886c800.ost_server_uuid 50 [ 4404.810299] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d538886c800.ost_server_uuid in IDLE state after 0 sec [ 4413.929646] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4413.936059] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.940058] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4413.945018] LustreError: Skipped 1 previous similar message [ 4413.950845] Lustre: Skipped 1 previous similar message [ 4414.268928] Lustre: server umount lustre-OST0001 complete [ 4419.050312] LustreError: 142647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.055675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.068034] LustreError: 142647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4419.098283] Lustre: Skipped 1 previous similar message [ 4423.793566] Lustre: server umount lustre-OST0000 complete [ 4429.282722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.299260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4429.309454] Lustre: Skipped 4 previous similar messages [ 4433.926753] Lustre: server umount lustre-MDT0000 complete [ 4434.400795] LustreError: 138899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.418983] LustreError: 138899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4437.679979] LustreError: 138881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776168 with bad export cookie 3504535904112399066 [ 4437.680931] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.698744] LustreError: 138881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4438.071691] Lustre: server umount lustre-MDT0001 complete [ 4459.409983] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4462.136983] Key type lgssc unregistered [ 4462.415852] LNet: 145273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4462.423177] LNetError: 145273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4462.437862] LNet: Removed LNI 192.168.202.130@tcp [ 4463.570491] Key type .llcrypt unregistered [ 4463.573305] Key type ._llcrypt unregistered [ 4508.736558] Key type ._llcrypt registered [ 4508.738746] Key type .llcrypt registered [ 4508.851846] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4525.361812] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4526.483605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4526.638409] alg: No test for adler32 (adler32-zlib) [ 4527.844864] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 4528.122329] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4529.847248] Key type lgssc registered [ 4531.200593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4567.876655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4567.950528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.295072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.320201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.399115] Lustre: lustre-MDT0000: new disk, initializing [ 4569.504855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.533351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4574.750858] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4586.685674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.880268] Lustre: 149446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4586.963209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4586.969655] Lustre: Skipped 1 previous similar message [ 4587.055125] Lustre: lustre-MDT0001: new disk, initializing [ 4587.138653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4587.170222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4587.181565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4592.881977] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4604.781680] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4613.047871] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4620.978094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4621.315866] Lustre: lustre-OST0000: new disk, initializing [ 4621.320612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4621.330681] Lustre: 151416:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4621.475763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.170615] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4630.102822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4630.120921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4630.342277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4640.457261] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4645.864833] Lustre: Failing over lustre-OST0000 [ 4645.870903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -19 [ 4645.879643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.907109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4647.912962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4647.932760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.953957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.143154] Lustre: server umount lustre-OST0000 complete [ 4669.215941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.623792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4669.664331] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4675.107896] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4677.390255] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4681.961554] Lustre: Failing over lustre-MDT0000 [ 4682.533680] Lustre: server umount lustre-MDT0000 complete [ 4684.768160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.787815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.471284] LustreError: 149451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4690.490527] LustreError: 149451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4695.575302] LustreError: 149452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.595028] LustreError: 149452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4700.694252] LustreError: 149453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4700.715329] LustreError: 149453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4701.155277] Lustre: 147659:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790776416/real 1790776416] req@000000005e2ea61e x1877765006605440/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790776432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4701.191374] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.080578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.272561] LustreError: 151009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.294965] LustreError: 151009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4711.397768] LustreError: 149451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4711.420994] LustreError: 149451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4712.418948] LustreError: 147658:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000f594495 x1877765006609664/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4712.706624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4712.742592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.773420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4716.813886] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4717.105596] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4718.070691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4718.119428] Lustre: lustre-OST0000: Recovery over after 0:43, of 3 clients 3 recovered and 0 were evicted. [ 4718.132675] Lustre: Skipped 1 previous similar message [ 4718.182537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4728.293342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4728.296776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.326489] Lustre: Skipped 2 previous similar messages [ 4728.344338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.350313] Lustre: Skipped 1 previous similar message [ 4731.177036] Lustre: server umount lustre-OST0000 complete [ 4738.536098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4738.540557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.553060] LustreError: Skipped 1 previous similar message [ 4738.556649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4738.582762] Lustre: Skipped 2 previous similar messages [ 4741.958969] Lustre: server umount lustre-MDT0000 complete [ 4743.648895] LustreError: 149453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.685945] LustreError: 149453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4745.718380] LustreError: 151415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790776476 with bad export cookie 11516192640242523808 [ 4745.722293] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.735696] LustreError: 151415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4746.227780] Lustre: server umount lustre-MDT0001 complete [ 4758.989526] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 4762.595273] Key type lgssc unregistered [ 4762.941946] LNet: 155074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4762.965863] LNetError: 155074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4764.021204] LNet: Removed LNI 192.168.202.130@tcp [ 4765.411869] Key type .llcrypt unregistered [ 4765.415483] Key type ._llcrypt unregistered [ 4810.854198] Key type ._llcrypt registered [ 4810.856621] Key type .llcrypt registered [ 4810.955574] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4812.076615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4812.115546] alg: No test for adler32 (adler32-zlib) [ 4813.229312] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 4813.471028] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 4815.151291] Key type lgssc registered [ 4816.600109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4823.891155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4823.907037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.314664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.332213] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4835.811580] LustreError: 156616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.837532] LustreError: 156616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4839.957579] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4840.930785] LustreError: 156615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.092898] LustreError: 156644:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4844.102733] LustreError: 156644:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4844.110479] LustreError: 156644:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4844.404626] Lustre: server umount lustre-MDT0000 complete [ 4859.928637] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 4871.121300] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 4907.690936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.908773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4907.931159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4907.992577] Lustre: lustre-MDT0000: new disk, initializing [ 4908.048137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.059162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4912.678599] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4924.511727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.645914] Lustre: 159947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4924.720570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4924.724521] Lustre: Skipped 1 previous similar message [ 4924.814512] Lustre: lustre-MDT0001: new disk, initializing [ 4924.951355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4925.002887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4925.011094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4929.986685] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4941.851114] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4951.007105] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4959.846893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4960.126916] Lustre: lustre-OST0000: new disk, initializing [ 4960.140900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4960.151559] Lustre: 161916:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4960.254945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4965.511076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4965.523325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4965.667450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4967.909231] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 4980.484362] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4986.042169] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4988.182208] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:58:37 (1790776717) [ 5262.817312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5262.821219] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.839585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5262.859035] Lustre: Skipped 1 previous similar message [ 5267.937381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.948519] Lustre: Skipped 4 previous similar messages [ 5268.414480] Lustre: server umount lustre-MDT0000 complete [ 5272.333492] LustreError: 161915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777003 with bad export cookie 12137805993674984866 [ 5272.333754] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.349156] LustreError: 161915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5273.056858] LustreError: 159955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.069211] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.083485] Lustre: Skipped 1 previous similar message [ 5273.086248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5277.088159] LustreError: 162635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.095261] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5278.788707] Lustre: server umount lustre-MDT0001 complete [ 5289.605997] Lustre: server umount lustre-OST0000 complete [ 5301.702315] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 5314.856054] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5357.242793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.518163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5357.548876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5357.622551] Lustre: lustre-MDT0000: new disk, initializing [ 5357.717573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.744348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5363.187799] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5378.763361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.891512] Lustre: 166597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5378.941593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5378.945318] Lustre: Skipped 1 previous similar message [ 5379.012707] Lustre: lustre-MDT0001: new disk, initializing [ 5379.096979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5379.147834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5379.169550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5384.154357] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5395.713965] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5402.943395] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5410.808977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.126461] Lustre: lustre-OST0000: new disk, initializing [ 5411.129660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5411.134978] Lustre: 168568:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5411.218174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.872083] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5420.619350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5420.623487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5420.761612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5427.970483] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5433.494078] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5433.802587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5438.902241] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5439.071346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5441.505026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.511060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5441.536215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.752335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.761989] Lustre: Skipped 3 previous similar messages [ 5455.327174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5455.573950] Lustre: server umount lustre-OST0000 complete [ 5460.961326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5460.965814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.982163] Lustre: Skipped 1 previous similar message [ 5466.884989] Lustre: server umount lustre-MDT0000 complete [ 5467.108194] LustreError: 166603:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.126994] LustreError: 166603:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5471.422825] LustreError: 166590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777202 with bad export cookie 12137805993675143696 [ 5471.438105] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.762339] Lustre: server umount lustre-MDT0001 complete [ 5482.148776] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:06:50 (1790777210) [ 5501.355091] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 5511.877836] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5553.050687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.653399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5553.720458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5553.903757] Lustre: lustre-MDT0000: new disk, initializing [ 5553.983855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.996672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5560.290882] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5572.632333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.833547] Lustre: 173327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5573.184197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5578.744194] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5589.601254] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.672892] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5602.705575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.950979] Lustre: lustre-OST0000: new disk, initializing [ 5602.953826] Lustre: Skipped 1 previous similar message [ 5602.957956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5602.962203] Lustre: Skipped 2 previous similar messages [ 5602.967190] Lustre: 175305:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5609.544086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5609.554432] Lustre: Skipped 1 previous similar message [ 5609.563547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5609.712908] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5609.742119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5619.678512] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5630.450924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5630.454798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.474882] Lustre: Skipped 1 previous similar message [ 5630.479192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5630.489490] Lustre: Skipped 4 previous similar messages [ 5634.477528] Lustre: server umount lustre-OST0000 complete [ 5639.651971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5639.665177] LustreError: Skipped 1 previous similar message [ 5639.671822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.688212] Lustre: Skipped 1 previous similar message [ 5645.797587] LustreError: 173335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.825305] LustreError: 173335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5646.079752] Lustre: server umount lustre-MDT0000 complete [ 5650.097233] LustreError: 173321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777381 with bad export cookie 12137805993675144578 [ 5650.097774] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5650.108026] LustreError: 173321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5650.669645] Lustre: server umount lustre-MDT0001 complete [ 5661.736695] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:09:50 (1790777390) [ 5681.866771] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 5693.190083] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5731.049865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.396701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5731.449383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5731.548864] Lustre: lustre-MDT0000: new disk, initializing [ 5731.677256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.685905] Lustre: Skipped 2 previous similar messages [ 5731.711365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5736.313091] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5747.638462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.794897] Lustre: 179630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5748.091721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5752.958406] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5762.731556] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5769.625325] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5778.453909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.805418] Lustre: 181610:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5786.245347] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5786.445042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5786.784674] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5800.178942] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5812.192516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5812.194506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.209257] LustreError: Skipped 1 previous similar message [ 5812.213165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5812.234732] Lustre: Skipped 2 previous similar messages [ 5812.246412] Lustre: Skipped 4 previous similar messages [ 5814.214662] Lustre: server umount lustre-OST0000 complete [ 5824.464330] Lustre: server umount lustre-MDT0000 complete [ 5827.552551] LustreError: 179643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.563892] LustreError: 179643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5828.373408] LustreError: 179622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777559 with bad export cookie 12137805993675145663 [ 5828.378155] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.390172] LustreError: 179622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5828.703544] Lustre: server umount lustre-MDT0001 complete [ 5840.523719] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 5843.421630] Key type lgssc unregistered [ 5843.693217] LNet: 183465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5843.703320] LNetError: 183465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5843.718307] LNet: Removed LNI 192.168.202.130@tcp [ 5844.611052] Key type .llcrypt unregistered [ 5844.612248] Key type ._llcrypt unregistered [ 5872.752576] Key type ._llcrypt registered [ 5872.754166] Key type .llcrypt registered [ 5872.882485] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 5874.182029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5874.295680] alg: No test for adler32 (adler32-zlib) [ 5875.432744] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 5875.839944] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 5877.595995] Key type lgssc registered [ 5878.957880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5887.879450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5887.899624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.413124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5893.781811] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5899.752858] LustreError: 184897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5899.772589] LustreError: 184897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5904.393692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.837271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5909.371465] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5917.826876] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.182221] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5933.093959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.572862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.612931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5940.778855] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 5950.015430] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5970.400508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5970.402089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5970.432281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.447604] Lustre: Skipped 1 previous similar message [ 5975.523488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5975.526589] Lustre: Skipped 1 previous similar message [ 5976.080113] Lustre: server umount lustre-OST0000 complete [ 5980.641789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.642403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.655281] Lustre: Skipped 2 previous similar messages [ 5980.673900] Lustre: Skipped 1 previous similar message [ 5985.763529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.779173] Lustre: Skipped 1 previous similar message [ 5987.070598] Lustre: server umount lustre-MDT0000 complete [ 5990.885305] LustreError: 187402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.912480] LustreError: 187402:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5991.095733] LustreError: 184878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777722 with bad export cookie 8184127720994675247 [ 5991.117435] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.126259] LustreError: 184878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5991.807477] Lustre: server umount lustre-MDT0001 complete [ 6005.778322] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 6009.735238] Key type lgssc unregistered [ 6010.126394] LNet: 188457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6010.148872] LNetError: 188457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6011.176146] LNet: Removed LNI 192.168.202.130@tcp [ 6012.057465] Key type .llcrypt unregistered [ 6012.059601] Key type ._llcrypt unregistered [ 6037.805816] Key type ._llcrypt registered [ 6037.808662] Key type .llcrypt registered [ 6037.913303] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6038.898085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6039.128110] alg: No test for adler32 (adler32-zlib) [ 6040.237468] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 6040.506792] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 6042.223525] Key type lgssc registered [ 6043.166872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6050.320728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6050.333867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.637104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6055.886991] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6062.053310] LustreError: 189890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.078703] LustreError: 189890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6064.495802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.823379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.637355] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6076.712089] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6082.549805] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6090.301540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.589278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6096.760418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6097.035555] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6104.832673] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6115.297134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6115.311474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.323278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6115.347109] Lustre: Skipped 1 previous similar message [ 6116.305513] Lustre: server umount lustre-OST0000 complete [ 6124.554785] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d5385d36000.ost_server_uuid 50 [ 6134.876318] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6151.136503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.157253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.160124] Lustre: Skipped 1 previous similar message [ 6155.299457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6155.312996] Lustre: Skipped 1 previous similar message [ 6155.906595] Lustre: server umount lustre-MDT0000 complete [ 6160.363440] LustreError: 189885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.389613] LustreError: 189885:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6160.779120] LustreError: 189870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790777891 with bad export cookie 1078672397679735782 [ 6160.787883] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6160.803024] LustreError: 189870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6161.274045] Lustre: server umount lustre-MDT0001 complete [ 6171.895871] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:18:20 (1790777900) [ 6179.627944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6179.998360] LustreError: 193807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.068874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.371347] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6185.451424] LustreError: 193808:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.484156] LustreError: 193808:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6189.540033] LustreError: 193807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.158534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.506100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6200.301241] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6207.842311] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6214.127706] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6220.284052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6220.562341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6224.688319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6226.680505] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6234.843100] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6241.560353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.891398] Lustre: lustre-OST0001: new disk, initializing [ 6241.898483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6241.912447] Lustre: 196649:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6242.020523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6249.602181] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6250.641741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6250.660127] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6250.794130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6260.507136] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6265.824781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.829675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.836110] Lustre: Skipped 1 previous similar message [ 6265.839679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.310787] Lustre: server umount lustre-OST0000 complete [ 6271.478936] LustreError: 195631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.525846] LustreError: 195631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6277.595681] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6277.827807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6282.752327] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6282.977204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6291.954606] LustreError: 195631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.977488] LustreError: 195631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6302.179817] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6302.182904] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6302.185497] LustreError: Skipped 1 previous similar message [ 6302.188051] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6302.198083] Lustre: Skipped 2 previous similar messages [ 6302.212802] Lustre: Skipped 4 previous similar messages [ 6303.571908] Lustre: server umount lustre-OST0001 complete [ 6308.320464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6308.331741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.419676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6312.431158] Lustre: Skipped 2 previous similar messages [ 6314.683255] Lustre: server umount lustre-MDT0000 complete [ 6318.498744] LustreError: 193788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778049 with bad export cookie 1078672397679736853 [ 6318.509087] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.510721] LustreError: 193788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6318.834305] Lustre: server umount lustre-MDT0001 complete [ 6341.933618] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:21:11 (1790778071) [ 6349.010827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6349.294433] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6349.326869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6349.512716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6354.794909] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6363.279600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.360155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6363.391749] Lustre: 200094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6367.569579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6374.849314] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6380.542874] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6387.076067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.196946] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6393.713445] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6395.516346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6401.713617] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6408.626905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6408.676490] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6414.833391] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6422.533773] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6426.080614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6426.087522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.114023] Lustre: Skipped 1 previous similar message [ 6426.122023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6431.667565] Lustre: server umount lustre-OST0000 complete [ 6433.317244] LustreError: 201945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.344915] LustreError: 201945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6450.145467] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6450.150531] LustreError: Skipped 1 previous similar message [ 6450.156179] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.172290] Lustre: Skipped 1 previous similar message [ 6455.324492] Lustre: server umount lustre-OST0001 complete [ 6466.023182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6466.063709] Lustre: Skipped 7 previous similar messages [ 6467.676048] Lustre: server umount lustre-MDT0000 complete [ 6473.114225] LustreError: 200087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778204 with bad export cookie 1078672397679738260 [ 6473.124252] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6473.138040] LustreError: 200087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6473.620518] Lustre: server umount lustre-MDT0001 complete [ 6500.040297] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:23:49 (1790778229) [ 6517.447723] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 6528.179583] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 6563.512486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.728879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6563.742162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6563.786786] Lustre: lustre-MDT0000: new disk, initializing [ 6563.833026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6563.840485] Lustre: Skipped 3 previous similar messages [ 6563.850407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6567.684475] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6579.138654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.375891] Lustre: 208191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6579.501614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6579.509641] Lustre: Skipped 1 previous similar message [ 6579.613439] Lustre: lustre-MDT0001: new disk, initializing [ 6579.813248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6579.841465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6585.568168] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6598.691402] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6605.732560] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6613.679506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6614.020168] Lustre: lustre-OST0000: new disk, initializing [ 6614.028569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6614.034893] Lustre: 210161:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6620.229914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6620.239259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6620.375127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6621.764316] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6631.316305] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6637.288950] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6637.474440] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6642.252894] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6642.562686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6646.245545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6646.249255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6646.250827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6646.257406] Lustre: Skipped 4 previous similar messages [ 6646.265372] LustreError: Skipped 2 previous similar messages [ 6650.940779] Lustre: server umount lustre-OST0000 complete [ 6661.722631] Lustre: server umount lustre-MDT0000 complete [ 6665.542176] LustreError: 208181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778396 with bad export cookie 1078672397679739821 [ 6665.558496] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6666.060076] Lustre: server umount lustre-MDT0001 complete [ 6674.705167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.144340] LustreError: 212379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6675.163441] LustreError: 212379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6680.318740] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6690.836883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6696.685672] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6705.577822] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6713.229222] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6721.447157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.049934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.063676] Lustre: Skipped 4 previous similar messages [ 6730.134964] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6737.668523] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6741.915119] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6746.532380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6751.363729] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6751.601311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6755.809557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6755.823322] Lustre: Skipped 2 previous similar messages [ 6755.825820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6755.833294] LustreError: Skipped 1 previous similar message [ 6759.772742] Lustre: server umount lustre-OST0000 complete [ 6765.821185] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6770.860392] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6800.155826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.719624] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6816.954736] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6819.032762] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6821.090582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6821.118887] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6821.129806] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6821.154826] Lustre: Skipped 1 previous similar message [ 6826.464519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6826.467086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6826.471895] LustreError: Skipped 1 previous similar message [ 6826.495923] Lustre: Skipped 8 previous similar messages [ 6829.919503] Lustre: server umount lustre-OST0000 complete [ 6841.164391] Lustre: server umount lustre-MDT0000 complete [ 6841.824673] LustreError: 212374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6841.855417] LustreError: 212374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6845.842182] LustreError: 212361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778576 with bad export cookie 1078672397679740703 [ 6845.860147] LustreError: 212361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6845.875471] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6846.165722] Lustre: server umount lustre-MDT0001 complete [ 6855.246485] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:29:44 (1790778584) [ 6862.493506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.963474] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6878.598709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.043600] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6895.909468] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6903.623758] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6912.426342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.252859] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6929.159140] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6933.794373] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6934.106700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6939.670511] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6939.880429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6949.635862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6949.842785] Lustre: lustre-OST0001: new disk, initializing [ 6949.846681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6949.851534] Lustre: 221751:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6956.608789] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 6957.739383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6957.748699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6957.860907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6967.255436] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6973.033394] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6973.361390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6979.510607] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6979.809640] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6983.648025] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.651560] LustreError: Skipped 1 previous similar message [ 6983.659645] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.681465] Lustre: Skipped 6 previous similar messages [ 6987.907270] Lustre: server umount lustre-OST0001 complete [ 6995.084848] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6995.365963] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7000.094544] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7000.273819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7029.794773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7030.011035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7030.020370] Lustre: Skipped 5 previous similar messages [ 7035.391562] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7035.405798] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7035.407305] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7036.196122] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7044.475768] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7046.403875] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7054.801795] Lustre: server umount lustre-OST0001 complete [ 7077.951403] Lustre: server umount lustre-OST0000 complete [ 7084.004608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7084.012049] Lustre: Skipped 11 previous similar messages [ 7096.287253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7096.513709] Lustre: server umount lustre-MDT0000 complete [ 7101.113458] LustreError: 218354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790778832 with bad export cookie 1078672397679741669 [ 7101.127279] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.133428] LustreError: 218354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7107.603928] Lustre: server umount lustre-MDT0001 complete [ 7134.343246] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:34:23 (1790778863) [ 7140.953288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7141.284237] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7141.314265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7145.907684] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7155.271758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.344208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7155.373655] Lustre: 226994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7155.443952] LustreError: 227004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.471843] LustreError: 227004:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 7160.390328] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7167.845893] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7173.317408] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7181.060642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7181.249808] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7189.092348] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7200.718219] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7213.587412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.702105] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7222.256867] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7230.199578] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7231.977533] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7236.833330] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7237.011524] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7241.936364] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7242.134844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7248.421403] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d5387491000.ost_server_uuid 50 [ 7250.324346] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d5387491000.ost_server_uuid in IDLE state after 0 sec [ 7251.507096] Lustre: Permanently deactivating lustre-OST0001 [ 7251.523359] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7255.118433] Lustre: Permanently reactivating lustre-OST0001 [ 7261.676703] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.683590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7261.703147] Lustre: Skipped 6 previous similar messages [ 7261.732033] LustreError: Skipped 6 previous similar messages [ 7265.262527] Lustre: server umount lustre-OST0001 complete [ 7275.975181] Lustre: server umount lustre-OST0000 complete [ 7287.388147] Lustre: server umount lustre-MDT0000 complete [ 7292.282444] LustreError: 226984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779023 with bad export cookie 1078672397679742957 [ 7292.292965] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.297183] LustreError: 226984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7292.863085] Lustre: server umount lustre-MDT0001 complete [ 7320.843806] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:37:29 (1790779049) [ 7329.932157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.388982] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7330.467737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7330.482215] Lustre: Skipped 5 previous similar messages [ 7335.316892] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7344.800458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.901162] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7344.955616] Lustre: 233690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7350.221543] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7358.557776] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7364.524791] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7372.078230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7372.363384] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7372.397396] Lustre: Permanently deactivating lustre-OST0000 [ 7372.410309] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7379.011301] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7380.733822] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7392.009748] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.417620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.574524] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7407.337626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7408.356497] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7416.988311] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7418.795717] Lustre: Permanently reactivating lustre-OST0000 [ 7418.809224] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7418.815888] Lustre: Skipped 2 previous similar messages [ 7425.093498] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7425.102586] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 00000000b030b973, cur 1790779156 deadline 1790779140 last 1790779110 [ 7425.180695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7444.287735] Lustre: server umount lustre-OST0001 complete [ 7463.391417] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7463.535965] Lustre: server umount lustre-OST0000 complete [ 7481.311237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7481.510474] Lustre: server umount lustre-MDT0000 complete [ 7485.040259] LustreError: 233683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779215 with bad export cookie 1078672397679744609 [ 7485.048317] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7485.050136] LustreError: 233683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7485.336862] Lustre: server umount lustre-MDT0001 complete [ 7497.201078] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7499.851938] Key type lgssc unregistered [ 7500.096751] LNet: 238571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7500.104438] LNetError: 238571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7500.116508] LNet: Removed LNI 192.168.202.130@tcp [ 7501.152287] Key type .llcrypt unregistered [ 7501.154710] Key type ._llcrypt unregistered [ 7524.801812] Key type ._llcrypt registered [ 7524.804820] Key type .llcrypt registered [ 7524.908951] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7525.958806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7526.034508] alg: No test for adler32 (adler32-zlib) [ 7527.055974] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 7527.326478] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7528.967207] Key type lgssc registered [ 7530.041387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7541.182817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7541.195764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.652788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.167047] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7552.993935] LustreError: 240051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.005553] LustreError: 240051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7555.179854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.265360] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7555.284922] Lustre: 240039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7555.291315] Lustre: Permanently deactivating lustre-MDT0001 [ 7555.301283] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7555.549771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7558.426300] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7558.431735] LustreError: 240082:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7558.441621] LustreError: 240082:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7558.448824] LustreError: 240082:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7559.324724] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7566.843927] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.217296] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7578.142570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.356436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.393888] LustreError: 241890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7580.408368] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7583.805582] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7585.766777] LustreError: 241890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.827199] LustreError: 242040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7587.855842] LustreError: 242040:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7587.860841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7591.334288] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7592.945991] LustreError: 241890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7592.994888] LustreError: 241890:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7599.575886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7599.959682] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7605.273187] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7605.289209] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7608.278616] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7617.791319] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7620.974892] Lustre: Permanently reactivating lustre-MDT0001 [ 7620.997831] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7621.004945] Lustre: Skipped 2 previous similar messages [ 7625.787464] Lustre: lustre-MDT0001: Client 58196dbd-5943-442e-a305-340ce83c4943 (at 192.168.202.30@tcp) reconnecting [ 7633.125772] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7635.312908] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7641.982857] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7642.313935] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7646.607932] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7646.885241] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7649.424697] Lustre: Permanently deactivating lustre-MDT0001 [ 7649.453490] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7649.456328] Lustre: Skipped 2 previous similar messages [ 7655.565351] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7668.192716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7668.201238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.238212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7668.708980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.726717] Lustre: Skipped 2 previous similar messages [ 7672.775991] Lustre: server umount lustre-MDT0000 complete [ 7673.830123] LustreError: 240051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.872201] LustreError: 240051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7677.138266] LustreError: 240033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779408 with bad export cookie 8007407928007792620 [ 7677.153652] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7677.156830] LustreError: 240033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7677.413440] Lustre: server umount lustre-MDT0001 complete [ 7684.838472] Lustre: server umount lustre-OST0000 complete [ 7697.689553] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:43:47 (1790779427) [ 7709.375689] Lustre: server umount lustre-OST0001 complete [ 7716.499979] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 7726.629254] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7763.687537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.911659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7763.921714] Lustre: Skipped 2 previous similar messages [ 7763.938783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7763.992332] Lustre: lustre-MDT0000: new disk, initializing [ 7764.046072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7764.064148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7767.977867] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7779.003773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.127719] Lustre: 248333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7779.191317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7779.194788] Lustre: Skipped 1 previous similar message [ 7779.262387] Lustre: lustre-MDT0001: new disk, initializing [ 7779.371533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7779.406656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7779.419835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7784.148232] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7793.282305] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7799.151545] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7807.233053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.543533] Lustre: lustre-OST0000: new disk, initializing [ 7807.546838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7807.558747] Lustre: 250300:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7807.663687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7813.391971] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7815.719116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7815.725547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7815.857803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7822.026821] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7824.345568] LustreError: 248339:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7833.157514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7833.323479] Lustre: lustre-OST0001: new disk, initializing [ 7833.326632] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7833.331609] Lustre: 251414:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7834.375124] LustreError: 248339:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7834.412210] LustreError: 248340:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7839.196870] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7841.366296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7841.382873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7841.458941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7844.503085] LustreError: 248340:0:(mdt_handler.c:3275:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7848.068549] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7849.298706] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7851.061815] Lustre: lustre-OST0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 7851.074532] Lustre: Skipped 3 previous similar messages [ 7852.000048] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7852.005158] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7852.025715] LustreError: Skipped 1 previous similar message [ 7852.041099] Lustre: Skipped 1 previous similar message [ 7856.157473] Lustre: lustre-OST0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 7856.167662] Lustre: Skipped 2 previous similar messages [ 7861.322766] Lustre: lustre-OST0001: Not available for connect from 192.168.202.30@tcp (stopping) [ 7861.332106] Lustre: Skipped 2 previous similar messages [ 7865.313456] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7865.439349] Lustre: server umount lustre-OST0001 complete [ 7866.398394] LustreError: 250286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7882.727435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7882.737761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.765538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7882.770779] Lustre: Skipped 2 previous similar messages [ 7887.332796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7887.342138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.496551] Lustre: server umount lustre-OST0000 complete [ 7892.973486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7892.997062] Lustre: Skipped 2 previous similar messages [ 7898.499100] Lustre: server umount lustre-MDT0000 complete [ 7903.200305] LustreError: 248345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.227892] LustreError: 248345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7903.677984] LustreError: 248325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779634 with bad export cookie 8007407928007795434 [ 7903.681268] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7903.681624] LustreError: 248325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7910.377327] Lustre: server umount lustre-MDT0001 complete [ 7927.566337] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 7930.687719] Key type lgssc unregistered [ 7930.968938] LNet: 253447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7930.973212] LNetError: 253447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7930.983226] LNet: Removed LNI 192.168.202.130@tcp [ 7931.920849] Key type .llcrypt unregistered [ 7931.922673] Key type ._llcrypt unregistered [ 7973.043963] Key type ._llcrypt registered [ 7973.050728] Key type .llcrypt registered [ 7973.174648] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 7974.066428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7974.088290] alg: No test for adler32 (adler32-zlib) [ 7975.063971] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 7975.221299] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 7976.871370] Key type lgssc registered [ 7977.925199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7984.317611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7984.339928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.540991] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7985.551437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7985.658906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7989.655470] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 7997.241899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.361685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7997.402068] Lustre: 255671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7997.679123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8001.839371] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8008.964733] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8015.229442] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8022.417733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8022.685790] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8022.880619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8029.888668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8030.404432] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8038.845076] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8096.226035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8096.228876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.239450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.738957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8096.755433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8096.771457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8099.493055] Lustre: server umount lustre-OST0000 complete [ 8104.238736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8117.420210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8117.766350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8124.896910] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8131.781839] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8136.108749] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8136.128762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8136.137646] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8172.002711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.006366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8172.028762] Lustre: Skipped 1 previous similar message [ 8172.049165] Lustre: Skipped 1 previous similar message [ 8177.123017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8177.141145] Lustre: Skipped 1 previous similar message [ 8177.597556] Lustre: server umount lustre-OST0000 complete [ 8187.361699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.362458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8187.406955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.172675] Lustre: server umount lustre-MDT0000 complete [ 8192.656463] LustreError: 255663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790779923 with bad export cookie 13619186897743038181 [ 8192.665159] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8192.678211] LustreError: 255663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8192.963082] Lustre: server umount lustre-MDT0001 complete [ 8203.894586] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 8206.168591] Key type lgssc unregistered [ 8206.476151] LNet: 261096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8206.492113] LNetError: 261096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8206.505328] LNet: Removed LNI 192.168.202.130@tcp [ 8207.175463] Key type .llcrypt unregistered [ 8207.177151] Key type ._llcrypt unregistered [ 8231.297365] Key type ._llcrypt registered [ 8231.299489] Key type .llcrypt registered [ 8231.374468] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 8232.268438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8232.419893] alg: No test for adler32 (adler32-zlib) [ 8233.467462] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 8233.788839] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 8235.463171] Key type lgssc registered [ 8236.518673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8244.916732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8244.935190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.309116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8250.770154] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8256.491240] LustreError: 262531:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.512476] LustreError: 262531:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8260.780252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.284392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.562522] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8274.853216] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8282.929982] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.331123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.932125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.264106] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8306.354469] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8316.630788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8347.623413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.624158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8347.635974] LustreError: Skipped 1 previous similar message [ 8347.645073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8347.654016] Lustre: Skipped 1 previous similar message [ 8347.682214] Lustre: Skipped 1 previous similar message [ 8350.448677] Lustre: server umount lustre-OST0000 complete [ 8357.860868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.872980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8357.879124] Lustre: Skipped 1 previous similar message [ 8361.006516] Lustre: server umount lustre-MDT0000 complete [ 8362.978153] LustreError: 262532:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8363.013910] LustreError: 262532:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8365.451525] LustreError: 262513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780096 with bad export cookie 17799029435564071753 [ 8365.452347] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8365.459480] LustreError: 262513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8365.970744] Lustre: server umount lustre-MDT0001 complete [ 8377.768699] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 8380.669219] Key type lgssc unregistered [ 8380.959987] LNet: 266855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8380.966539] LNetError: 266855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8380.985922] LNet: Removed LNI 192.168.202.130@tcp [ 8381.703708] Key type .llcrypt unregistered [ 8381.707340] Key type ._llcrypt unregistered [ 8401.644647] Key type ._llcrypt registered [ 8401.650440] Key type .llcrypt registered [ 8401.816183] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 8402.962918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8403.160203] alg: No test for adler32 (adler32-zlib) [ 8404.410750] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 8404.864581] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 8406.551341] Key type lgssc registered [ 8407.481058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8418.651525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8418.680977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.187478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.946873] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8430.566379] LustreError: 268094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8430.596575] LustreError: 268094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8435.499927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8435.689946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8435.851804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8440.452458] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8448.462818] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8454.423922] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8462.088414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8462.486181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8465.586981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8469.429183] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8477.917451] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8496.617052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8496.621441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.631944] LustreError: Skipped 1 previous similar message [ 8496.641683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.646804] Lustre: Skipped 1 previous similar message [ 8500.095495] Lustre: server umount lustre-OST0000 complete [ 8506.849070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8506.868730] Lustre: Skipped 1 previous similar message [ 8506.878635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8510.655232] Lustre: server umount lustre-MDT0000 complete [ 8511.969952] LustreError: 268094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8511.983748] LustreError: 268094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8514.543952] LustreError: 268074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780245 with bad export cookie 17073255348077258829 [ 8514.551578] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8514.562337] LustreError: 268074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8514.963798] Lustre: server umount lustre-MDT0001 complete [ 8527.494155] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 8530.195804] Key type lgssc unregistered [ 8530.538685] LNet: 271770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8530.545587] LNetError: 271770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8531.621733] LNet: Removed LNI 192.168.202.130@tcp [ 8532.514210] Key type .llcrypt unregistered [ 8532.516712] Key type ._llcrypt unregistered [ 8560.199918] Key type ._llcrypt registered [ 8560.204275] Key type .llcrypt registered [ 8560.337244] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 8561.552302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8561.697045] alg: No test for adler32 (adler32-zlib) [ 8563.087859] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 8563.411141] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 8565.111217] Key type lgssc registered [ 8566.404635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8576.787773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8576.827795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.334924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.078624] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8588.768864] LustreError: 273206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8588.784116] LustreError: 273206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8593.903776] LustreError: 273205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.077519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8596.738030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.223441] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8611.730259] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8617.480913] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8623.966376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.371622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8630.837307] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8639.885287] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8648.947971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8677.862922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8677.874069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.895519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.905428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8679.912841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8679.921349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8679.935729] Lustre: Skipped 1 previous similar message [ 8681.948316] Lustre: server umount lustre-OST0000 complete [ 8689.120604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8689.128838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.143565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8691.831273] Lustre: server umount lustre-MDT0000 complete [ 8695.267570] LustreError: 273206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.279347] LustreError: 273206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8695.469829] LustreError: 273187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780426 with bad export cookie 18119590757436102595 [ 8695.475641] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8695.486444] LustreError: 273187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8695.818332] Lustre: server umount lustre-MDT0001 complete [ 8708.618338] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 8711.360728] Key type lgssc unregistered [ 8711.814572] LNet: 277534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8711.825558] LNetError: 277534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8711.836705] LNet: Removed LNI 192.168.202.130@tcp [ 8712.598304] Key type .llcrypt unregistered [ 8712.601195] Key type ._llcrypt unregistered [ 8732.446488] Key type ._llcrypt registered [ 8732.449205] Key type .llcrypt registered [ 8732.564895] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 8733.718590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8733.829993] alg: No test for adler32 (adler32-zlib) [ 8735.008214] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 8735.190473] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 8736.855350] Key type lgssc registered [ 8738.361405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8746.536934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8746.566508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.239462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8752.662556] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8758.753182] LustreError: 278802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8758.798833] LustreError: 278802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8762.356881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8762.702421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8767.108560] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8774.262924] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8780.270575] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8787.190614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8787.520295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8793.665361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8794.482229] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8802.248697] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8816.937547] Lustre: server umount lustre-OST0000 complete [ 8824.288733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8824.292721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.301159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8824.335645] Lustre: Skipped 1 previous similar message [ 8829.412846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.420461] Lustre: Skipped 2 previous similar messages [ 8831.709130] Lustre: server umount lustre-MDT0000 complete [ 8834.528775] LustreError: 278792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8834.548478] LustreError: 278792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8835.553057] Lustre: 278089:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790780550/real 1790780550] req@00000000ff7f73a5 x1877769417944320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790780566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8835.557098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.569147] Lustre: 278089:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8835.613252] LustreError: 278804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8836.449513] LustreError: 278769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780567 with bad export cookie 14818137775020020881 [ 8836.455901] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8836.459953] LustreError: 278769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8836.889753] Lustre: server umount lustre-MDT0001 complete [ 8850.020328] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 8853.016779] Key type lgssc unregistered [ 8853.419073] LNet: 282501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8853.428802] LNetError: 282501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8853.445343] LNet: Removed LNI 192.168.202.130@tcp [ 8854.344668] Key type .llcrypt unregistered [ 8854.346777] Key type ._llcrypt unregistered [ 8865.719402] Key type ._llcrypt registered [ 8865.722852] Key type .llcrypt registered [ 8865.858714] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8882.798397] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 8899.559510] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 8900.521899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8900.550430] alg: No test for adler32 (adler32-zlib) [ 8902.045100] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 8902.497753] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 8904.279434] Key type lgssc registered [ 8905.283934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8947.065166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8947.094067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.499289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8948.550185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8948.619408] Lustre: lustre-MDT0000: new disk, initializing [ 8948.766702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.816971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8953.948480] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8966.955497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.247788] Lustre: 286072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8967.322079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8967.325812] Lustre: Skipped 1 previous similar message [ 8967.417814] Lustre: lustre-MDT0001: new disk, initializing [ 8967.538260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8967.614338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8967.626527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8972.994920] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 8984.287888] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8990.787993] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8998.786649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.270561] Lustre: lustre-OST0000: new disk, initializing [ 8999.274203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8999.285823] Lustre: 288039:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8999.453755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9007.198562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9007.214237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9007.368825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9007.757690] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9020.344173] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9025.924464] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9026.221785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9032.393822] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9032.600650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9038.319025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9038.326447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.327974] LustreError: Skipped 1 previous similar message [ 9038.332370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9038.343152] Lustre: Skipped 1 previous similar message [ 9038.371582] Lustre: Skipped 1 previous similar message [ 9040.764885] Lustre: server umount lustre-OST0000 complete [ 9044.448330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9044.454687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.462826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.546260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.547251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.845758] Lustre: server umount lustre-MDT0000 complete [ 9055.733053] LustreError: 286063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780786 with bad export cookie 16028606959445469430 [ 9055.741371] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.757413] LustreError: 286063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9056.413941] Lustre: server umount lustre-MDT0001 complete [ 9067.898820] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:06:36 (1790780796) [ 9075.807909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9076.279500] LustreError: 290453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.302020] LustreError: 290453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9076.380204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9081.829773] LustreError: 290454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9081.985604] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9086.945099] LustreError: 290453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.908216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.069791] LustreError: 290454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.377441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9096.661150] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9103.856913] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9110.490808] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9117.940204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.647165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9126.558561] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9136.427958] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.360363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9147.364702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.380501] LustreError: Skipped 1 previous similar message [ 9147.396218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9147.405701] Lustre: Skipped 1 previous similar message [ 9147.931804] Lustre: server umount lustre-OST0000 complete [ 9154.043499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9154.053425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.063365] Lustre: Skipped 1 previous similar message [ 9154.070088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9154.080710] Lustre: Skipped 1 previous similar message [ 9159.009658] Lustre: server umount lustre-MDT0000 complete [ 9162.724843] LustreError: 290453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9162.741000] LustreError: 290453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9163.987045] LustreError: 292298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790780894 with bad export cookie 16028606959445470312 [ 9164.007156] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9164.015809] LustreError: 292298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9164.410415] Lustre: server umount lustre-MDT0001 complete [ 9176.861707] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 9179.618763] Key type lgssc unregistered [ 9180.048852] LNet: 293991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9180.064880] LNetError: 293991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9180.090445] LNet: Removed LNI 192.168.202.130@tcp [ 9181.070892] Key type .llcrypt unregistered [ 9181.073485] Key type ._llcrypt unregistered [ 9210.231248] Key type ._llcrypt registered [ 9210.233428] Key type .llcrypt registered [ 9210.337951] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 9226.931557] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 9228.152737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9228.206647] alg: No test for adler32 (adler32-zlib) [ 9229.233369] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 9229.486464] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 9231.137403] Key type lgssc registered [ 9232.486301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9281.409939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9281.439571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.811690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9282.872429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9283.001739] Lustre: lustre-MDT0000: new disk, initializing [ 9283.096965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9283.118039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9288.057826] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9298.091479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.201117] Lustre: 297836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9298.243787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9298.247513] Lustre: Skipped 1 previous similar message [ 9298.315584] Lustre: lustre-MDT0001: new disk, initializing [ 9298.403851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9298.438398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9298.454298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9302.776953] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9314.088931] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9320.000252] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9326.443801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9326.766236] Lustre: lustre-OST2710: new disk, initializing [ 9326.768780] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9326.771996] Lustre: 299802:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9326.820607] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9332.118704] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9332.806808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9332.824793] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9332.964398] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9341.916736] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9350.549261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9350.818481] Lustre: lustre-OST03e8: new disk, initializing [ 9350.825359] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9350.836638] Lustre: 300866:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9351.001301] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9360.208306] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9360.624586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9360.638425] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9360.800651] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9369.763694] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9377.305752] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9377.635343] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9383.727334] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9383.862795] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9390.435245] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9390.647920] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9394.897804] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9395.120816] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9400.812572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9400.822900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.849850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9401.766267] Lustre: server umount lustre-MDT0000 complete [ 9401.835278] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.454289] LustreError: 298630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781137 with bad export cookie 16521980538763895345 [ 9406.470775] LustreError: 298630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9406.480645] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9406.969790] Lustre: server umount lustre-MDT0001 complete [ 9421.617898] Lustre: server umount lustre-OST2710 complete [ 9422.303113] Lustre: 295621:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790781137/real 1790781137] req@000000007582068a x1877769935983104/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790781153 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9422.337655] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.352930] Lustre: Skipped 2 previous similar messages [ 9425.854802] Lustre: server umount lustre-OST03e8 complete [ 9445.323485] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [ 9456.892376] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 9500.966473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.141261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9501.160502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9501.223735] Lustre: lustre-MDT0000: new disk, initializing [ 9501.288926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.307357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9506.472284] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9519.834389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9520.029667] Lustre: 306117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9520.123379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9520.130849] Lustre: Skipped 1 previous similar message [ 9520.269374] Lustre: lustre-MDT0001: new disk, initializing [ 9520.464993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9520.566704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9520.587061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9525.480055] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9536.792434] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9543.368844] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9550.766084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.110057] Lustre: lustre-OST0000: new disk, initializing [ 9551.117680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9551.128361] Lustre: 308089:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9551.231042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.131311] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9559.641887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9559.661074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9559.768545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9569.550562] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9575.429947] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9575.761289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9581.516855] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9581.743181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9585.633583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.638827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9585.651930] Lustre: Skipped 1 previous similar message [ 9585.666460] Lustre: Skipped 1 previous similar message [ 9589.916653] Lustre: server umount lustre-OST0000 complete [ 9595.877871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.903869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.904115] Lustre: Skipped 1 previous similar message [ 9595.925418] Lustre: Skipped 1 previous similar message [ 9600.992195] LustreError: 307285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9601.002215] LustreError: 307285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9601.044095] Lustre: server umount lustre-MDT0000 complete [ 9605.091805] LustreError: 306129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9605.517339] LustreError: 306109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781336 with bad export cookie 16521980538763897025 [ 9605.533636] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9605.543915] LustreError: 306109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9606.022327] Lustre: server umount lustre-MDT0001 complete [ 9616.462240] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:15:45 (1790781345) [ 9618.235312] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9620.101797] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:15:49 (1790781349) [ 9623.838569] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 9652.307812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.619428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9652.666760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9652.869374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9658.237685] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9669.189539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9669.359705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9669.446731] Lustre: 311948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9674.961430] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9697.796775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.173437] Lustre: Denying initial registration attempt from nid 192.168.202.130@tcp, specified as failover [ 9698.193179] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9698.217962] LustreError: 313404:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9698.223389] LustreError: 313404:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9698.244537] LustreError: 313404:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9698.356761] Lustre: server umount lustre-OST0000 complete [ 9698.360346] LustreError: 313404:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9706.976620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9706.989963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9712.098826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9712.128471] Lustre: Skipped 2 previous similar messages [ 9721.312363] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9721.634912] Lustre: server umount lustre-MDT0000 complete [ 9722.337831] LustreError: 311953:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.359949] LustreError: 311953:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9726.201845] LustreError: 311939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781457 with bad export cookie 16521980538763897907 [ 9726.216816] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9726.222316] LustreError: 311939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9726.720379] Lustre: server umount lustre-MDT0001 complete [ 9742.248653] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 9745.285683] Key type lgssc unregistered [ 9745.597812] LNet: 314526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9745.611559] LNetError: 314526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9745.638417] LNet: Removed LNI 192.168.202.130@tcp [ 9746.616348] Key type .llcrypt unregistered [ 9746.618155] Key type ._llcrypt unregistered [ 9760.714144] Key type ._llcrypt registered [ 9760.717352] Key type .llcrypt registered [ 9760.829231] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 9762.182777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9762.191695] alg: No test for adler32 (adler32-zlib) [ 9763.349301] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 9763.570509] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [ 9765.295301] Key type lgssc registered [ 9766.772855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9821.085899] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [ 9831.671216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9831.705319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9833.004215] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9833.048596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9833.174393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9838.941790] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9849.286862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.405280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9849.487683] Lustre: 317306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9849.784456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9855.149528] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9864.915379] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9873.141940] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9894.198662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.621848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9894.914367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9902.980537] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [ 9913.423701] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9920.994032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.003228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9921.023764] Lustre: Skipped 1 previous similar message [ 9921.041440] Lustre: Skipped 1 previous similar message [ 9926.116732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9926.122356] Lustre: Skipped 1 previous similar message [ 9930.208046] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9930.571199] Lustre: server umount lustre-OST0000 complete [ 9936.358323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9936.397637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.417467] Lustre: Skipped 1 previous similar message [ 9941.472436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.485695] Lustre: Skipped 1 previous similar message [ 9942.238212] Lustre: server umount lustre-MDT0000 complete [ 9946.746867] LustreError: 317299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781677 with bad export cookie 12141167866673077612 [ 9946.760572] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9946.761409] LustreError: 317299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9947.387985] Lustre: server umount lustre-MDT0001 complete [ 9962.779593] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [ 9966.994954] Key type lgssc unregistered [ 9967.590646] LNet: 320885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9967.597193] LNetError: 320885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9967.618738] LNet: Removed LNI 192.168.202.130@tcp [ 9969.162575] Key type .llcrypt unregistered [ 9969.167488] Key type ._llcrypt unregistered [ 9999.231928] Key type ._llcrypt registered [ 9999.240064] Key type .llcrypt registered [ 9999.390142] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [10001.192898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10001.390930] alg: No test for adler32 (adler32-zlib) [10002.760499] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [10003.051636] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [10004.711278] Key type lgssc registered [10006.090610] Lustre: Echo OBD driver; http://www.lustre.org/ [10015.785744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10015.812498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.443678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10022.722171] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10028.003183] LustreError: 322320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.021643] LustreError: 322320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10031.552547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10032.039261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10036.619962] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10044.650547] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10052.310280] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10064.304884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10065.004260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10073.087369] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10083.880835] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10101.849649] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [10103.269435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10103.274977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10103.295843] Lustre: Skipped 1 previous similar message [10103.315208] Lustre: Skipped 2 previous similar messages [10103.941600] Lustre: server umount lustre-MDT0000 complete [10106.917356] LustreError: 323677:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.385544] LustreError: 323334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10108.432134] LustreError: 323334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10109.407354] LustreError: 322302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781840 with bad export cookie 6549066307422798250 [10109.444283] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10112.029568] LustreError: 322316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.111043] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [10113.512351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10113.535176] Lustre: Skipped 1 previous similar message [10114.532942] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10114.537224] Lustre: Skipped 1 previous similar message [10116.321825] Lustre: server umount lustre-MDT0001 complete [10125.594719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10135.285951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10154.469923] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5ae2f51bb06a5d95 [10154.504015] Lustre: MGC192.168.202.130@tcp: Connection restored to 0@lo (at 0@lo) [10154.934612] LustreError: 324153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10154.965114] LustreError: 324153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10155.090643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10155.181444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10162.620330] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10164.197900] LustreError: 325935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.223251] LustreError: 325935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10169.323093] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10174.473538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10174.749352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10180.083944] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10180.628709] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10188.436092] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10190.470632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10196.709178] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10199.034909] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10205.674218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10205.699492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10205.701587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10211.071212] Lustre: server umount lustre-OST0000 complete [10215.904858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10215.909329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10215.928823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10215.928832] Lustre: Skipped 3 previous similar messages [10215.931747] Lustre: Skipped 2 previous similar messages [10221.635384] Lustre: server umount lustre-MDT0000 complete [10225.814086] LustreError: 322300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790781956 with bad export cookie 6549066307422805397 [10225.819302] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10225.831326] LustreError: 322300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10226.317956] Lustre: server umount lustre-MDT0001 complete [10240.493112] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [10243.585542] Key type lgssc unregistered [10243.917712] LNet: 328613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10243.928966] LNetError: 328613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10243.948866] LNet: Removed LNI 192.168.202.130@tcp [10244.784562] Key type .llcrypt unregistered [10244.790437] Key type ._llcrypt unregistered [10274.310320] Key type ._llcrypt registered [10274.315423] Key type .llcrypt registered [10274.439247] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [10276.161127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10276.207655] alg: No test for adler32 (adler32-zlib) [10277.330670] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [10277.691585] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [10279.415239] Key type lgssc registered [10280.862612] Lustre: Echo OBD driver; http://www.lustre.org/ [10289.755220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10289.786890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.361471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10296.975620] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10301.948303] LustreError: 330046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.990251] LustreError: 330046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10307.044595] LustreError: 330047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.554631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.024942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10313.835253] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10323.121150] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10330.170811] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10337.769152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10337.792693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10337.798987] Lustre: Skipped 1 previous similar message [10340.465860] Lustre: server umount lustre-MDT0000 complete [10342.882222] LustreError: 330046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.899222] LustreError: 330046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10344.558400] LustreError: 330027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782075 with bad export cookie 14358664290856627104 [10344.579476] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10344.908634] Lustre: server umount lustre-MDT0001 complete [10362.973707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10363.175585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10363.194160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10363.373552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10369.483316] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10378.898398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10379.037218] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10379.062306] Lustre: 332632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10379.419493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10384.381270] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10394.539873] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10400.792607] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10414.072579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.351102] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10414.358797] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10414.367879] LustreError: 334518:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10414.383300] LustreError: 334518:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10414.386624] LustreError: 334518:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10414.464255] Lustre: server umount lustre-OST0000 complete [10414.467383] LustreError: 334518:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10427.118221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10427.400924] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10427.738361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10433.126380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10434.224738] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10442.085674] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10451.939976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10452.124987] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10452.244345] Lustre: lustre-OST0001: new disk, initializing [10452.266835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10452.277860] Lustre: 336060:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10452.453748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10461.197903] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10462.392808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10462.417428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10462.575635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10469.810446] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10478.057232] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10478.058494] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.072651] LustreError: Skipped 1 previous similar message [10478.074480] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10478.109608] Lustre: Skipped 2 previous similar messages [10483.173993] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10483.191399] Lustre: Skipped 2 previous similar messages [10488.287208] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10488.305673] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10488.323986] Lustre: Skipped 1 previous similar message [10488.511861] Lustre: server umount lustre-OST0001 complete [10493.417378] LustreError: 336452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.437085] LustreError: 336452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10494.437086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10494.440858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.443490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10494.448026] Lustre: Skipped 1 previous similar message [10494.466471] Lustre: Skipped 1 previous similar message [10498.540621] LustreError: 336452:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.583121] LustreError: 336452:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10499.855750] Lustre: server umount lustre-OST0000 complete [10507.232704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10507.243723] LustreError: Skipped 1 previous similar message [10507.257080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.286264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10507.291949] Lustre: Skipped 2 previous similar messages [10510.446353] Lustre: server umount lustre-MDT0000 complete [10513.899422] LustreError: 334943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10513.924614] LustreError: 334943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10515.004043] LustreError: 332627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782245 with bad export cookie 14358664290856627657 [10515.010416] LustreError: 332627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10515.013931] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10515.711323] Lustre: server umount lustre-MDT0001 complete [10528.974640] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [10533.462498] Key type lgssc unregistered [10534.250351] LNet: 338022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10534.266200] LNetError: 338022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10534.337353] LNet: Removed LNI 192.168.202.130@tcp [10535.854881] Key type .llcrypt unregistered [10535.857147] Key type ._llcrypt unregistered [10610.110762] Key type ._llcrypt registered [10610.112943] Key type .llcrypt registered [10610.197362] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [10628.421377] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [10630.375486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10630.400189] alg: No test for adler32 (adler32-zlib) [10631.573888] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [10631.834081] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [10633.535243] Key type lgssc registered [10634.687837] Lustre: Echo OBD driver; http://www.lustre.org/ [10676.165779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10676.212769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10677.588856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10677.617126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10677.704429] Lustre: lustre-MDT0000: new disk, initializing [10677.784472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10677.800426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10682.278979] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10693.617319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10693.757595] Lustre: 342976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10693.808470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10693.813236] Lustre: Skipped 1 previous similar message [10693.883938] Lustre: lustre-MDT0001: new disk, initializing [10693.971431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10694.015826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10694.030453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10698.517194] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10709.658626] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10716.548570] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10724.033698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10724.344860] Lustre: lustre-OST0000: new disk, initializing [10724.350792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10724.360925] Lustre: 344944:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10724.463151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10730.737569] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10733.107333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10733.122231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10733.242744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10740.878352] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10746.068949] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10746.409863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10751.974230] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10752.218443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10759.151909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.155154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10759.173600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10759.184098] Lustre: Skipped 1 previous similar message [10760.406163] Lustre: server umount lustre-OST0000 complete [10764.256934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.270580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10764.272334] Lustre: Skipped 2 previous similar messages [10764.290810] Lustre: Skipped 1 previous similar message [10769.376484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10769.396827] Lustre: Skipped 1 previous similar message [10770.483225] Lustre: server umount lustre-MDT0000 complete [10774.424980] LustreError: 342967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782505 with bad export cookie 18446514848624820066 [10774.435931] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10774.445854] LustreError: 342967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10774.846738] Lustre: server umount lustre-MDT0001 complete [10784.621747] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:35:13 (1790782513) [10793.836433] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:35:22 (1790782522) [10803.999601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10804.406315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10809.323688] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10809.832530] LustreError: 347700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10809.858882] LustreError: 347700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10814.945608] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.456413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.699536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10823.072909] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10830.055187] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10836.261254] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10843.676393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.103194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10850.209261] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10857.552374] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10869.847959] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10871.643869] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10873.609825] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10875.561601] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10877.527969] Lustre: Failing over lustre-MDT0000 [10878.953746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.965177] Lustre: Skipped 1 previous similar message [10878.968034] LustreError: 347696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10878.993763] LustreError: 347696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10879.900805] Lustre: server umount lustre-MDT0000 complete [10884.068052] LustreError: 347696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.099275] LustreError: 347696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10889.186654] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10889.206343] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10894.304970] Lustre: 341187:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790782609/real 1790782609] req@00000000be1fa13f x1877771406087936/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790782625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10894.359144] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.426663] LustreError: 347695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.451637] LustreError: 347695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [10899.477031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.551202] LustreError: 341185:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000050f3f613 x1877771406092544/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10904.813211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10904.904584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10907.184690] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10910.192197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10910.236774] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10910.258674] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10910.294335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10921.057997] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10922.901818] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10926.653414] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10928.539068] Lustre: Failing over lustre-MDT0000 [10929.115587] Lustre: server umount lustre-MDT0000 complete [10930.658309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.670281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10930.671514] Lustre: Skipped 2 previous similar messages [10930.671651] LustreError: 350913:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.671658] LustreError: 350913:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10930.739414] LustreError: Skipped 1 previous similar message [10947.039369] Lustre: 341187:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790782661/real 1790782661] req@000000004f5d6ecb x1877771406108416/t0(0) o400->MGC192.168.202.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790782677 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10947.066592] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10948.778131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.536633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10957.602293] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10959.389302] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10962.325971] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [10962.959249] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10962.971880] Lustre: Skipped 2 previous similar messages [10963.014286] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10963.063581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10974.542994] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10976.398493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10988.517400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10988.521816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.527282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.582564] Lustre: Skipped 3 previous similar messages [10990.331261] Lustre: server umount lustre-MDT0000 complete [10993.650521] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.667204] LustreError: 347699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [10994.022236] LustreError: 351684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782724 with bad export cookie 18446514848624822810 [10994.033221] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10994.048962] LustreError: 351684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10994.342989] Lustre: server umount lustre-MDT0001 complete [11009.282996] Lustre: server umount lustre-OST0000 complete [11026.621899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.078513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11032.593573] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11042.470750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.920773] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11055.692069] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11061.875047] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11070.344143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11071.876580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11078.171658] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11086.336344] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11097.571043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11097.576091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.581777] LustreError: Skipped 1 previous similar message [11097.596439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.599081] Lustre: Skipped 1 previous similar message [11097.624393] Lustre: Skipped 3 previous similar messages [11102.683567] Lustre: server umount lustre-OST0000 complete [11107.812293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11112.741439] Lustre: server umount lustre-MDT0000 complete [11112.936109] LustreError: 354063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11112.970614] LustreError: 354063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11116.796785] LustreError: 354043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782847 with bad export cookie 18446514848624823531 [11116.811876] LustreError: 354043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11116.813713] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11117.198701] Lustre: server umount lustre-MDT0001 complete [11131.005273] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [11133.486221] Key type lgssc unregistered [11133.800193] LNet: 357596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11133.816747] LNetError: 357596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11133.852182] LNet: Removed LNI 192.168.202.130@tcp [11134.752487] Key type .llcrypt unregistered [11134.754938] Key type ._llcrypt unregistered [11165.137711] Key type ._llcrypt registered [11165.139799] Key type .llcrypt registered [11165.256849] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [11166.409025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11166.513799] alg: No test for adler32 (adler32-zlib) [11167.631842] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [11167.995746] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [11169.671218] Key type lgssc registered [11171.087971] Lustre: Echo OBD driver; http://www.lustre.org/ [11180.172322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11180.213722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.829509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11187.001385] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11191.270664] LustreError: 359145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.291864] LustreError: 359145:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11196.392706] LustreError: 359146:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.944246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11197.402945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11201.528971] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11210.516709] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11217.474632] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11224.515417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11224.839092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11226.875298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11231.805720] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11240.051938] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11253.729505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11253.750964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11253.780026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11257.840084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11257.845961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11259.433718] Lustre: server umount lustre-MDT0000 complete [11262.915743] LustreError: 360991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790782993 with bad export cookie 15793036758112857457 [11262.932402] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.946347] LustreError: 359140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.954046] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.993630] Lustre: Skipped 1 previous similar message [11263.004678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11263.017753] Lustre: Skipped 2 previous similar messages [11263.299728] Lustre: server umount lustre-MDT0001 complete [11269.646470] Lustre: server umount lustre-OST0000 complete [11283.908533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.709228] LustreError: 362873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11284.809818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11290.080674] LustreError: 362874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.047988] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11299.299313] LustreError: 362874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.319560] LustreError: 362874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11299.944567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.417349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11304.915317] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11313.115622] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11319.803891] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11326.946457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11327.582278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11329.659055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11335.198991] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11343.494419] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11356.640640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11356.641496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.651193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11356.673448] Lustre: Skipped 2 previous similar messages [11361.760352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11361.770618] Lustre: Skipped 4 previous similar messages [11362.786578] Lustre: server umount lustre-MDT0000 complete [11366.882574] LustreError: 362868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.904626] LustreError: 362868:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11367.436782] LustreError: 363608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783098 with bad export cookie 15793036758112858577 [11367.447146] LustreError: 363608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11367.448376] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11367.700504] Lustre: server umount lustre-MDT0001 complete [11382.477127] Lustre: server umount lustre-OST0000 complete [11397.813451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.200190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11403.233934] LustreError: 366786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11403.248570] LustreError: 366786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11403.807486] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11413.380290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.718736] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11426.176505] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11432.760345] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11441.675106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11442.334149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11442.338307] Lustre: Skipped 1 previous similar message [11443.394813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11450.544867] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11460.088680] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11469.291314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11469.305682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11469.316636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11469.332781] Lustre: Skipped 1 previous similar message [11471.367906] Lustre: server umount lustre-OST0000 complete [11475.424259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11475.441736] LustreError: Skipped 1 previous similar message [11479.529542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11479.558756] Lustre: Skipped 1 previous similar message [11481.566535] Lustre: server umount lustre-MDT0000 complete [11484.642548] LustreError: 366785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.672563] LustreError: 366785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11485.521037] LustreError: 367521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783216 with bad export cookie 15793036758112859592 [11485.530871] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11485.541694] LustreError: 367521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11485.848904] Lustre: server umount lustre-MDT0001 complete [11499.818249] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [11501.991935] Key type lgssc unregistered [11502.243521] LNet: 370319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11502.252669] LNetError: 370319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11502.265200] LNet: Removed LNI 192.168.202.130@tcp [11503.045961] Key type .llcrypt unregistered [11503.049294] Key type ._llcrypt unregistered [11533.997796] Key type ._llcrypt registered [11534.000955] Key type .llcrypt registered [11534.207117] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [11535.965818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11535.996119] alg: No test for adler32 (adler32-zlib) [11537.125838] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [11537.350477] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [11539.025077] Key type lgssc registered [11541.761327] Lustre: Echo OBD driver; http://www.lustre.org/ [11550.746193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11550.772107] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11550.778925] LustreError: 371774:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11550.813925] LustreError: 371774:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11550.825860] LustreError: 371774:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11550.837414] LustreError: 371774:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11550.846799] LustreError: 371774:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11559.959434] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11559.965294] LustreError: 372173:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11559.981102] LustreError: 372173:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11559.990765] LustreError: 372173:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11559.998092] LustreError: 372173:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11560.010675] LustreError: 372173:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11583.319967] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [11586.967495] Key type lgssc unregistered [11587.290833] LNet: 373175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11587.303357] LNetError: 373175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11587.336928] LNet: Removed LNI 192.168.202.130@tcp [11588.342717] Key type .llcrypt unregistered [11588.344994] Key type ._llcrypt unregistered [11607.464393] Key type ._llcrypt registered [11607.468370] Key type .llcrypt registered [11607.600151] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [11624.709313] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [11625.878611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11626.045647] alg: No test for adler32 (adler32-zlib) [11627.021750] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [11627.254544] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [11628.921142] Key type lgssc registered [11630.336629] Lustre: Echo OBD driver; http://www.lustre.org/ [11669.103483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11669.120802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.449069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11670.497495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11670.609543] Lustre: lustre-MDT0000: new disk, initializing [11670.734308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11670.750465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11675.725430] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11685.358257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11685.483643] Lustre: 376352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11685.532502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11685.535982] Lustre: Skipped 1 previous similar message [11685.593494] Lustre: lustre-MDT0001: new disk, initializing [11685.681302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11685.729494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11685.753124] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11690.792722] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11701.492709] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11707.475139] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11714.995636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11715.353127] Lustre: lustre-OST0000: new disk, initializing [11715.360313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11715.378164] Lustre: 378318:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11715.509530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11723.166726] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11723.448020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11723.470750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11723.637994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11736.996688] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11741.417275] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11741.827374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11748.588297] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11748.833467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11754.475196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11754.475623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11754.498427] LustreError: Skipped 1 previous similar message [11754.502377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11754.512950] Lustre: Skipped 1 previous similar message [11757.623917] Lustre: server umount lustre-OST0000 complete [11762.657341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11762.667729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11762.685094] Lustre: Skipped 1 previous similar message [11762.696090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11764.709626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11764.723561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11764.736667] Lustre: Skipped 1 previous similar message [11768.069659] Lustre: server umount lustre-MDT0000 complete [11772.443496] LustreError: 376345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783503 with bad export cookie 6249404641081399565 [11772.445605] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11772.460684] LustreError: 376345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11772.763803] Lustre: server umount lustre-MDT0001 complete [11781.502868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11783.302578] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:51:52 (1790783512) [11790.336747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11790.878775] LustreError: 380780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.898942] LustreError: 380780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11790.953793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11795.122849] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11796.448699] LustreError: 380781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.560972] LustreError: 380780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11803.943653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.228889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11808.758530] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11816.421560] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.682934] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11830.863438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11831.453338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11839.244710] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11851.074257] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11861.896929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11862.184512] Lustre: lustre-OST0001: new disk, initializing [11862.190360] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11862.197870] Lustre: 383614:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11862.310273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11869.338803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11869.348157] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11869.589371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11870.816799] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [11880.937060] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11885.024273] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11885.025736] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.036181] LustreError: Skipped 1 previous similar message [11885.050494] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11890.149684] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11890.159515] Lustre: Skipped 3 previous similar messages [11891.258847] Lustre: server umount lustre-OST0001 complete [11892.771736] LustreError: 382601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.388054] LustreError: 383915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11900.416191] LustreError: 383915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11907.552057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11907.561660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11907.578682] Lustre: Skipped 1 previous similar message [11907.587373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11910.641756] LustreError: 382599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.675328] LustreError: 382599:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11913.887826] Lustre: server umount lustre-OST0000 complete [11920.871404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11920.895817] Lustre: Skipped 2 previous similar messages [11925.153409] Lustre: server umount lustre-MDT0000 complete [11930.089761] LustreError: 380780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.111246] LustreError: 380780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11930.771169] LustreError: 380761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783661 with bad export cookie 6249404641081400447 [11930.772984] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11930.793979] LustreError: 380761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11937.674127] Lustre: server umount lustre-MDT0001 complete [11950.901334] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [11954.059730] Key type lgssc unregistered [11954.297342] LNet: 385604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11954.304773] LNetError: 385604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11955.365183] LNet: Removed LNI 192.168.202.130@tcp [11956.325345] Key type .llcrypt unregistered [11956.329416] Key type ._llcrypt unregistered [12015.444491] Key type ._llcrypt registered [12015.449310] Key type .llcrypt registered [12015.568660] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [12016.283828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12016.343074] alg: No test for adler32 (adler32-zlib) [12017.338254] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [12017.546209] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [12019.303468] Key type lgssc registered [12020.320116] Lustre: Echo OBD driver; http://www.lustre.org/ [12028.288906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12028.310994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12029.558403] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12029.578546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12029.741993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12034.677426] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12042.928521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12042.984375] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12043.007586] Lustre: 388421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12043.197062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12046.860249] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12054.379472] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12059.760713] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12063.713358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12063.719306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.742221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12064.225135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.250174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12064.258022] Lustre: Skipped 1 previous similar message [12067.950628] Lustre: server umount lustre-MDT0000 complete [12071.683210] LustreError: 388412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783802 with bad export cookie 5859577500754292253 [12071.687349] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12071.695183] LustreError: 388412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12072.220897] Lustre: server umount lustre-MDT0001 complete [12082.402494] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:56:51 (1790783811) [12089.185849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12089.560226] LustreError: 390932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.581719] LustreError: 390932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12089.607749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12094.351157] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12094.946456] LustreError: 390933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12100.066760] LustreError: 390932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.664487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12104.005816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12109.839811] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12119.053618] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12126.084399] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12135.035834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12135.297202] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12135.501213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12142.961749] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12151.761874] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12160.828935] Lustre: Permanently deactivating lustre-OST0000 [12160.856630] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12168.796600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12172.145613] LustreError: 393917:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12172.158987] LustreError: 393917:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12172.166162] LustreError: 393917:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12174.981670] LustreError: 393965:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12174.986890] LustreError: 393965:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12179.710408] Lustre: server umount lustre-OST0000 complete [12182.812979] LustreError: 394215:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12182.821177] LustreError: 394215:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12185.057664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.062830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12185.069904] Lustre: Skipped 1 previous similar message [12190.176820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12190.200773] Lustre: Skipped 2 previous similar messages [12195.298656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12195.307095] Lustre: Skipped 1 previous similar message [12199.391145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12199.625813] Lustre: server umount lustre-MDT0000 complete [12200.422735] LustreError: 390932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.450349] LustreError: 390932:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12203.358337] LustreError: 390912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790783934 with bad export cookie 5859577500754292834 [12203.364508] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12203.371961] LustreError: 390912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12213.895022] Lustre: server umount lustre-MDT0001 complete [12223.652959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12229.666674] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12233.774075] LustreError: 395377:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12241.388664] Lustre: 395716:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.130@tcp [12243.892789] Lustre: server umount MGS complete [12256.038597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12256.584253] LustreError: 396443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.613328] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12256.619182] Lustre: Skipped 1 previous similar message [12256.677330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12261.193702] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12266.989068] LustreError: 396444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.017710] LustreError: 396444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12270.647108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12270.973037] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12275.462797] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12285.235371] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12291.989810] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12294.435907] Lustre: Permanently reactivating lustre-OST0000 [12294.459759] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12294.466062] Lustre: Skipped 2 previous similar messages [12322.785119] LustreError: 398223:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12332.815511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12333.232711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12333.238206] Lustre: Skipped 1 previous similar message [12339.507899] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12348.915303] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12358.124910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12358.132355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.183492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12358.193354] Lustre: Skipped 1 previous similar message [12362.120386] Lustre: server umount lustre-OST0000 complete [12368.352804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12368.366630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12368.369602] LustreError: Skipped 1 previous similar message [12368.400497] Lustre: Skipped 1 previous similar message [12378.594162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12378.611439] Lustre: Skipped 5 previous similar messages [12381.152455] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12381.475698] Lustre: server umount lustre-MDT0000 complete [12383.712842] LustreError: 396970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12383.735872] LustreError: 396970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12385.654099] LustreError: 396423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784116 with bad export cookie 5859577500754293884 [12385.665664] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12385.669745] LustreError: 396423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12386.118557] Lustre: server umount lustre-MDT0001 complete [12398.629606] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [12401.581510] Key type lgssc unregistered [12401.836420] LNet: 400820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12401.843424] LNetError: 400820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12401.878922] LNet: Removed LNI 192.168.202.130@tcp [12402.626784] Key type .llcrypt unregistered [12402.632472] Key type ._llcrypt unregistered [12421.599965] Key type ._llcrypt registered [12421.601743] Key type .llcrypt registered [12421.686317] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [12437.377916] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [12438.623951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12438.776145] alg: No test for adler32 (adler32-zlib) [12439.901422] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [12440.161430] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [12441.888710] Key type lgssc registered [12443.201862] Lustre: Echo OBD driver; http://www.lustre.org/ [12486.633944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12486.672618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12487.924687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12487.946067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12488.005843] Lustre: lustre-MDT0000: new disk, initializing [12488.064452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12488.079418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12494.518531] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12508.336397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.654808] Lustre: 403994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12508.863881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12508.876861] Lustre: Skipped 1 previous similar message [12509.049914] Lustre: lustre-MDT0001: new disk, initializing [12509.305705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12509.350163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12509.358532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12515.355527] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12528.457691] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12535.632411] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12544.863440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12545.191741] Lustre: lustre-OST0000: new disk, initializing [12545.197761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12545.209787] Lustre: 405965:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12545.394420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12551.772137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12551.794171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12551.989851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12553.570553] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12566.888456] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12573.334579] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12573.617286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12580.619469] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12580.965360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12588.009049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12588.009395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12588.019645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12588.042955] Lustre: Skipped 1 previous similar message [12593.122662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12593.140122] Lustre: Skipped 2 previous similar messages [12598.239591] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12598.256981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12598.272764] Lustre: Skipped 1 previous similar message [12598.480867] Lustre: server umount lustre-OST0000 complete [12603.364902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12603.371519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12603.384407] Lustre: Skipped 1 previous similar message [12603.393476] Lustre: Skipped 1 previous similar message [12608.485153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12609.534217] Lustre: server umount lustre-MDT0000 complete [12614.359349] LustreError: 405964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784345 with bad export cookie 12943396934062224458 [12614.364109] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12614.368482] LustreError: 405964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12614.818793] Lustre: server umount lustre-MDT0001 complete [12626.111298] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12628.864711] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:05:57 (1790784357) [12637.879722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.991052] LustreError: 408429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.016102] LustreError: 408429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12639.095238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12644.338431] LustreError: 408430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.363392] LustreError: 408430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12645.971975] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12648.416457] LustreError: 408429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12653.555542] LustreError: 408430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12657.495456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.771192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12663.648454] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12673.804905] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12680.728996] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12689.647662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12690.107197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12697.096120] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12707.272378] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12709.794721] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12713.049964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12720.096325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12720.116872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12720.131540] Lustre: Skipped 2 previous similar messages [12726.532409] Lustre: server umount lustre-OST0000 complete [12734.433090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12734.445341] LustreError: Skipped 1 previous similar message [12734.454290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12734.467085] Lustre: Skipped 1 previous similar message [12737.357308] Lustre: server umount lustre-MDT0000 complete [12740.586508] LustreError: 409159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.631546] LustreError: 409159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12742.681828] LustreError: 408409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784473 with bad export cookie 12943396934062225340 [12742.682677] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12742.701367] LustreError: 408409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12743.513284] Lustre: server umount lustre-MDT0001 complete [12758.834530] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [12762.339701] Key type lgssc unregistered [12762.706593] LNet: 412156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12762.717651] LNetError: 412156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12763.772069] LNet: Removed LNI 192.168.202.130@tcp [12764.661086] Key type .llcrypt unregistered [12764.663874] Key type ._llcrypt unregistered [12792.459749] Key type ._llcrypt registered [12792.461411] Key type .llcrypt registered [12792.645916] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [12793.109389] Key type .llcrypt unregistered [12793.116311] Key type ._llcrypt unregistered [12818.605270] Key type ._llcrypt registered [12818.606710] Key type .llcrypt registered [12818.756129] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [12820.260770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12820.326770] alg: No test for adler32 (adler32-zlib) [12821.518339] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [12821.873633] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [12823.591257] Key type lgssc registered [12825.966802] Lustre: Echo OBD driver; http://www.lustre.org/ [12838.921224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12838.982217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.102648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12846.370371] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12850.657063] LustreError: 414385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.675374] LustreError: 414385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12855.781872] LustreError: 414386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12856.686530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12857.088324] LustreError: 414386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12857.140814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12862.433447] LustreError: 414385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12862.457098] LustreError: 414385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12864.458171] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12865.537225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12870.634655] LustreError: 414386:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.665337] LustreError: 414386:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12874.821512] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12880.870606] LustreError: 414385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12880.897924] LustreError: 414385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12885.975193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.472100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12891.423646] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [12901.348148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12901.359174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.373135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12901.884274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.890490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12901.916086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12902.774455] Lustre: server umount lustre-OST0000 complete [12906.978162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12906.986583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12907.007126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12907.548995] Lustre: server umount lustre-MDT0000 complete [12911.689406] LustreError: 416584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784642 with bad export cookie 3650198910054895129 [12911.694116] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12911.702300] LustreError: 416584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12912.143961] Lustre: server umount lustre-MDT0001 complete [12924.640163] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [12927.556686] Key type lgssc unregistered [12927.977856] LNet: 417533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12927.982582] LNetError: 417533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12928.007222] LNet: Removed LNI 192.168.202.130@tcp [12928.938091] Key type .llcrypt unregistered [12928.940571] Key type ._llcrypt unregistered [12958.360246] Key type ._llcrypt registered [12958.365333] Key type .llcrypt registered [12958.581867] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [12960.269560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12960.296880] alg: No test for adler32 (adler32-zlib) [12961.432746] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [12961.628452] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [12963.351228] Key type lgssc registered [12964.787724] Lustre: Echo OBD driver; http://www.lustre.org/ [12973.368951] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12973.389822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12989.025212] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12994.143616] LustreError: 418943:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.130@tcp: failed processing log, type 4: rc = -110 [13020.895321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13027.063687] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13036.575471] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13043.670824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13044.387549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13044.479158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13049.559873] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13049.829601] LustreError: 418969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.858433] LustreError: 418969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13054.957261] LustreError: 418968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13056.005650] LustreError: 419979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13056.031356] LustreError: 419979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13059.889346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13060.526661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13065.581224] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13075.042158] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13081.638399] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13090.789768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13090.795139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13090.804293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13091.304674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13091.307725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13091.333663] Lustre: Skipped 1 previous similar message [13091.713938] Lustre: server umount lustre-OST0000 complete [13096.098575] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13096.108561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13096.131395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13096.387261] Lustre: server umount lustre-MDT0000 complete [13096.440509] LustreError: 419982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13100.676962] LustreError: 418950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784831 with bad export cookie 17479949369300919822 [13100.683265] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13100.692499] LustreError: 418950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13101.009522] Lustre: server umount lustre-MDT0001 complete [13113.806524] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [13117.077714] Key type lgssc unregistered [13117.433627] LNet: 422510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13117.436381] LNetError: 422510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13117.448047] LNet: Removed LNI 192.168.202.130@tcp [13118.640609] Key type .llcrypt unregistered [13118.646064] Key type ._llcrypt unregistered [13148.068877] Key type ._llcrypt registered [13148.071763] Key type .llcrypt registered [13148.232297] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [13149.807359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13149.837820] alg: No test for adler32 (adler32-zlib) [13150.890308] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [13151.127845] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [13152.800203] Key type lgssc registered [13154.470034] Lustre: Echo OBD driver; http://www.lustre.org/ [13163.150573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13163.172620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13164.711287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13168.988536] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13174.265160] LustreError: 423945:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13174.297111] LustreError: 423945:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13179.075707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13179.360069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13179.578840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13184.434881] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13192.565942] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13198.251499] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13205.560841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13205.976592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13210.106266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13212.907571] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13222.744797] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13229.559964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13229.570040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13229.583497] Lustre: Skipped 1 previous similar message [13229.597217] Lustre: Skipped 2 previous similar messages [13230.639038] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.30@tcp (stopping) [13233.934789] Lustre: server umount lustre-MDT0000 complete [13234.659963] LustreError: 423944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13234.687307] LustreError: 423944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13235.739183] LustreError: 423941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.330270] LustreError: 423924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.30@tcp arrived at 1790784967 with bad export cookie 9131305107674939046 [13238.463876] LustreError: 423926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790784969 with bad export cookie 9131305107674938157 [13238.476821] LustreError: 423926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13238.484444] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13239.777986] LustreError: 423940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13239.784557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13239.807715] LustreError: 423940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13239.835970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13239.848528] Lustre: Skipped 1 previous similar message [13239.858193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13243.809156] LustreError: 424949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13243.833700] LustreError: 424949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13244.825522] Lustre: server umount lustre-OST0000 complete [13252.881601] Lustre: server umount lustre-MDT0001 complete [13264.851079] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [13269.055367] Key type lgssc unregistered [13269.472368] LNet: 427630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13269.483312] LNetError: 427630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13270.515433] LNet: Removed LNI 192.168.202.130@tcp [13271.818795] Key type .llcrypt unregistered [13271.820670] Key type ._llcrypt unregistered [13306.499644] Key type ._llcrypt registered [13306.502404] Key type .llcrypt registered [13306.616607] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [13308.422828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13308.486603] alg: No test for adler32 (adler32-zlib) [13309.881322] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [13310.314468] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [13312.103225] Key type lgssc registered [13313.644945] Lustre: Echo OBD driver; http://www.lustre.org/ [13321.985663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13322.005268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13323.581723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13328.760774] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13333.996871] LustreError: 429067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13334.017870] LustreError: 429067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13337.772151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13337.991621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13343.922913] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13352.328343] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13359.255460] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13367.729975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13368.050126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13369.117690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13376.119598] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13385.806084] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13392.870722] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13392.884842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13393.974906] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [13393.981734] Lustre: Skipped 1 previous similar message [13396.842592] Lustre: server umount lustre-MDT0001 complete [13397.992570] LustreError: 429064:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13398.010960] LustreError: 429064:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13403.116636] LustreError: 429062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13405.153262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13405.159426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13405.167222] Lustre: Skipped 1 previous similar message [13405.171959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13408.227464] LustreError: 429068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13408.234245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13408.244533] LustreError: 429068:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13409.531643] Lustre: server umount lustre-OST0000 complete [13413.346197] LustreError: 429795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13415.278666] Lustre: server umount lustre-MDT0000 complete [13430.531951] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [13433.091471] Key type lgssc unregistered [13433.368311] LNet: 432750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13433.387710] LNetError: 432750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13433.399994] LNet: Removed LNI 192.168.202.130@tcp [13434.245240] Key type .llcrypt unregistered [13434.247439] Key type ._llcrypt unregistered [13463.277956] Key type ._llcrypt registered [13463.279827] Key type .llcrypt registered [13463.357942] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [13480.473887] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [13481.649600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13481.752218] alg: No test for adler32 (adler32-zlib) [13482.779763] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [13482.960371] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [13484.600471] Key type lgssc registered [13485.811816] Lustre: Echo OBD driver; http://www.lustre.org/ [13530.035363] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [13552.464306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13552.502345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13553.930406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13554.008901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13554.152214] Lustre: lustre-MDT0000: new disk, initializing [13554.290692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13554.322077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13559.834199] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13578.932079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13579.329824] Lustre: lustre-OST0000: new disk, initializing [13579.333995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13579.340409] Lustre: Skipped 1 previous similar message [13579.346512] Lustre: 437804:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13579.411774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13585.586772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13585.595296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13585.651994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13586.982359] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13599.352289] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13618.310730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13618.382587] Lustre: 436784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13618.421292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13618.470361] Lustre: lustre-MDT0001: new disk, initializing [13618.567637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13618.591857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13618.598150] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13622.966290] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13632.873653] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13633.082769] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13638.842882] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13639.111673] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13646.309616] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13646.330396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13648.963081] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.30@tcp (stopping) [13648.967058] Lustre: Skipped 1 previous similar message [13650.806413] Lustre: server umount lustre-MDT0001 complete [13656.550539] LustreError: 436794:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13656.570274] LustreError: 436794:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13661.673699] LustreError: 439148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13661.692051] LustreError: 439148:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13666.787128] LustreError: 436792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13666.820460] LustreError: 436792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13671.904482] LustreError: 439148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13671.908031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13671.927957] Lustre: Skipped 1 previous similar message [13671.931911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13676.281589] Lustre: server umount lustre-OST0000 complete [13677.026987] LustreError: 436797:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13686.931593] Lustre: server umount lustre-MDT0000 complete [13703.132940] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [13705.904755] Key type lgssc unregistered [13706.226716] LNet: 441559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13706.236579] LNetError: 441559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13706.257702] LNet: Removed LNI 192.168.202.130@tcp [13707.139729] Key type .llcrypt unregistered [13707.142678] Key type ._llcrypt unregistered [13755.282062] Key type ._llcrypt registered [13755.284221] Key type .llcrypt registered [13755.424872] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [13755.765316] Key type .llcrypt unregistered [13755.766983] Key type ._llcrypt unregistered [13779.577943] Key type ._llcrypt registered [13779.579580] Key type .llcrypt registered [13779.894175] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [13781.560856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13781.600559] alg: No test for adler32 (adler32-zlib) [13783.327820] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [13783.706106] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [13785.504485] Key type lgssc registered [13787.592344] Lustre: Echo OBD driver; http://www.lustre.org/ [13825.876914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [13825.930679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13827.236961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13827.261986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13827.327716] Lustre: lustre-MDT0000: new disk, initializing [13827.400453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13827.416445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13831.616657] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13843.060699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13843.199536] Lustre: 444952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13843.279090] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13843.286382] Lustre: Skipped 1 previous similar message [13843.405121] Lustre: lustre-MDT0001: new disk, initializing [13843.530610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13843.577650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13843.593534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13847.792396] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13858.555921] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13864.769724] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13873.661972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13874.012411] Lustre: lustre-OST0000: new disk, initializing [13874.016771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13874.024641] Lustre: 446920:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13874.133693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13879.368221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13879.379293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13879.483993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13880.727648] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13891.724459] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13899.746220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13899.768204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13899.794686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13904.430295] Lustre: server umount lustre-MDT0000 complete [13909.008805] LustreError: 444947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785639 with bad export cookie 1051210505516699832 [13909.017331] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13909.025970] LustreError: 444947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13909.314788] Lustre: server umount lustre-MDT0001 complete [13923.722330] Lustre: server umount lustre-OST0000 complete [13935.725268] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:27:44 (1790785664) [13944.971706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13945.662600] LustreError: 448949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13945.687202] LustreError: 448949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13945.813673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13950.564458] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13950.947400] LustreError: 448950:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13956.080669] LustreError: 448949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13961.200438] LustreError: 448950:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13962.072694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13962.702132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13968.353248] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [13977.971314] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13985.855577] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14008.311947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14008.730129] Lustre: Found index 0 for lustre-OST0000, updating log [14009.042356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14016.184394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14016.458458] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [14027.581803] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14036.968338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14036.984252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14036.999021] Lustre: Skipped 2 previous similar messages [14037.014341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14037.024368] Lustre: Skipped 3 previous similar messages [14038.117084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14038.119643] Lustre: Skipped 2 previous similar messages [14041.789484] Lustre: server umount lustre-OST0000 complete [14060.002348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14060.013729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14060.049906] Lustre: Skipped 1 previous similar message [14060.073372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14064.160763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14064.169244] Lustre: Skipped 2 previous similar messages [14072.287710] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14072.444214] Lustre: server umount lustre-MDT0000 complete [14072.607230] LNet: 1 local NIs in recovery (showing 1): 192.168.202.130@tcp [14075.372186] LustreError: 448949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14075.399501] LustreError: 448949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14076.145507] LustreError: 448931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790785807 with bad export cookie 1051210505516701225 [14076.159475] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14076.160883] LustreError: 448931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14076.871443] Lustre: server umount lustre-MDT0001 complete [14087.723207] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:30:16 (1790785816) [14714.261421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14714.980639] LustreError: 455365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14715.094187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14720.482591] LustreError: 455366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14720.500618] LustreError: 455366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14721.158469] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [14724.595203] LustreError: 455365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14730.734281] LustreError: 455366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14730.751398] LustreError: 455366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14732.087901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14732.490792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14737.489689] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [14745.578547] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14752.790610] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14774.185539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14774.510266] Lustre: Found index 0 for lustre-OST0000, updating log [14774.926759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14776.101862] LNet: 1 local NIs in recovery (showing 1): 192.168.202.130@tcp [14783.081276] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [14791.999440] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14795.370567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14795.382912] Lustre: Skipped 1 previous similar message [14795.388482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14795.393534] Lustre: Skipped 4 previous similar messages [14800.852948] Lustre: server umount lustre-OST0000 complete [14805.351869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14805.360888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14805.366647] Lustre: Skipped 1 previous similar message [14807.526642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14807.540713] Lustre: Skipped 4 previous similar messages [14811.617674] LustreError: 457734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14811.627966] LustreError: 457734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14811.652362] Lustre: server umount lustre-MDT0000 complete [14815.892258] LustreError: 455346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786546 with bad export cookie 1051210505516702303 [14815.892853] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14815.902324] LustreError: 455346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14816.364757] Lustre: server umount lustre-MDT0001 complete [14836.511126] LNet: 1 local NIs in recovery (showing 1): 192.168.202.130@tcp [14849.706214] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [14859.819869] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [14903.933440] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:43:53 (1790786633) [14925.792964] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [14929.752972] Key type lgssc unregistered [14930.426494] LNet: 461978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14930.434118] LNetError: 461978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15037.991711] LNet: Removed LNI 192.168.202.130@tcp [15038.828248] Key type .llcrypt unregistered [15038.830296] Key type ._llcrypt unregistered [15058.517721] Key type ._llcrypt registered [15058.523845] Key type .llcrypt registered [15058.702720] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [15060.546508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15060.581038] alg: No test for adler32 (adler32-zlib) [15061.603270] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [15061.825199] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [15063.511385] Key type lgssc registered [15064.989953] Lustre: Echo OBD driver; http://www.lustre.org/ [15069.258470] LNet: Added LNI 192.168.202.130@tcp100 [8/256/0/180] [15070.032592] LNet: Added LNI 192.168.202.130@tcp101 [8/256/0/180] [15070.767696] LNet: Added LNI 192.168.202.130@tcp102 [8/256/0/180] [15071.716593] LNet: Added LNI 192.168.202.130@tcp103 [8/256/0/180] [15072.540872] LNet: Added LNI 192.168.202.130@tcp104 [8/256/0/180] [15073.570122] LNet: Added LNI 192.168.202.130@tcp105 [8/256/0/180] [15074.562208] LNet: Added LNI 192.168.202.130@tcp106 [8/256/0/180] [15075.281858] LNet: Added LNI 192.168.202.130@tcp107 [8/256/0/180] [15076.525728] LNet: Added LNI 192.168.202.130@tcp108 [8/256/0/180] [15077.899466] LNet: Added LNI 192.168.202.130@tcp109 [8/256/0/180] [15078.891047] LNet: Added LNI 192.168.202.130@tcp110 [8/256/0/180] [15079.904774] LNet: Added LNI 192.168.202.130@tcp111 [8/256/0/180] [15081.100477] LNet: Added LNI 192.168.202.130@tcp112 [8/256/0/180] [15082.023499] LNet: Added LNI 192.168.202.130@tcp113 [8/256/0/180] [15083.083508] LNet: Added LNI 192.168.202.130@tcp114 [8/256/0/180] [15084.086321] LNet: Added LNI 192.168.202.130@tcp115 [8/256/0/180] [15085.239888] LNet: Added LNI 192.168.202.130@tcp116 [8/256/0/180] [15086.340062] LNet: Added LNI 192.168.202.130@tcp117 [8/256/0/180] [15087.760587] LNet: Added LNI 192.168.202.130@tcp118 [8/256/0/180] [15088.967629] LNet: Added LNI 192.168.202.130@tcp119 [8/256/0/180] [15090.032878] LNet: Added LNI 192.168.202.130@tcp120 [8/256/0/180] [15091.186796] LNet: Added LNI 192.168.202.130@tcp121 [8/256/0/180] [15092.264895] LNet: Added LNI 192.168.202.130@tcp122 [8/256/0/180] [15093.425918] LNet: Added LNI 192.168.202.130@tcp123 [8/256/0/180] [15094.530696] LNet: Added LNI 192.168.202.130@tcp124 [8/256/0/180] [15095.777622] LNet: Added LNI 192.168.202.130@tcp125 [8/256/0/180] [15096.984543] LNet: Added LNI 192.168.202.130@tcp126 [8/256/0/180] [15097.963272] LNet: Added LNI 192.168.202.130@tcp127 [8/256/0/180] [15098.839949] LNet: Added LNI 192.168.202.130@tcp128 [8/256/0/180] [15099.683414] LNet: Added LNI 192.168.202.130@tcp129 [8/256/0/180] [15100.503457] LNet: Added LNI 192.168.202.130@tcp130 [8/256/0/180] [15101.487261] LNet: Added LNI 192.168.202.130@tcp131 [8/256/0/180] [15102.366915] LNet: Added LNI 192.168.202.130@tcp132 [8/256/0/180] [15103.258581] LNet: Added LNI 192.168.202.130@tcp133 [8/256/0/180] [15104.101818] LNet: Added LNI 192.168.202.130@tcp134 [8/256/0/180] [15105.156699] LNet: Added LNI 192.168.202.130@tcp135 [8/256/0/180] [15112.608198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15112.626595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15113.810466] Lustre: 465026:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [15113.903759] LNetError: 465048:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15113.909389] LNetError: Protocol error connecting to 192.168.202.130@tcp114 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15113.920252] LNetError: Skipped 1 previous similar message [15114.085036] LNet: 8 local NIs in recovery (showing 8): 192.168.202.130@tcp120, 192.168.202.130@tcp121, 192.168.202.130@tcp124, 192.168.202.130@tcp128, 192.168.202.130@tcp129, 192.168.202.130@tcp126, 192.168.202.130@tcp127, 192.168.202.130@tcp122 [15114.108827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15114.147233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15114.303335] Lustre: lustre-MDT0000: new disk, initializing [15114.380131] Lustre: 465082:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [15114.390400] Lustre: 465082:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15114.411821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15114.422459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15116.237250] LNetError: 462472:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15116.251271] LNetError: 462472:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 24 previous similar messages [15116.267871] LNetError: Protocol error connecting to 192.168.202.130@tcp121 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15116.290440] LNetError: Skipped 16 previous similar messages [15119.273838] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15120.356615] LNetError: 462475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15120.367771] LNetError: 462475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15120.378347] LNetError: Protocol error connecting to 192.168.202.130@tcp121 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15128.548456] LNetError: 465057:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15128.562233] LNetError: Protocol error connecting to 192.168.202.130@tcp121 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15129.985183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15130.223724] Lustre: 465043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15130.299891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15130.304077] Lustre: Skipped 1 previous similar message [15130.377916] Lustre: lustre-MDT0001: new disk, initializing [15130.419419] Lustre: 465844:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [15130.427517] Lustre: 465844:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15130.540189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15130.577340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15130.597072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15135.805529] Lustre: 466078:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [15135.812852] Lustre: 466078:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15136.079265] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15143.779226] LNetError: 465052:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15143.790625] LNetError: Protocol error connecting to 192.168.202.130@tcp117 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15147.396948] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15151.973028] LNetError: 465057:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15151.984354] LNetError: 465057:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15151.998430] LNetError: Protocol error connecting to 192.168.202.130@tcp124 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15152.009631] LNetError: Skipped 8 previous similar messages [15153.518521] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15160.846804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15161.116692] Lustre: lustre-OST0000: new disk, initializing [15161.122038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15161.135754] Lustre: 467018:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15161.251174] Lustre: 467023:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.130@tcp: can't add NID 192.168.202.130@tcp130: rc = -75 [15161.263958] Lustre: 467023:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15161.332501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15167.101617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15167.118153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15167.268234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15168.302905] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15168.361841] LNetError: 462474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15168.378037] LNetError: 462474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15168.390359] LNetError: Protocol error connecting to 192.168.202.130@tcp117 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15168.418923] LNetError: Skipped 13 previous similar messages [15174.751076] LNet: 1 local NIs in recovery (showing 1): 192.168.202.130@tcp121 [15178.910862] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15187.938451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15187.940324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15187.947208] LustreError: Skipped 1 previous similar message [15187.958858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15187.972392] Lustre: Skipped 1 previous similar message [15187.998707] Lustre: Skipped 1 previous similar message [15192.945173] Lustre: server umount lustre-OST0000 complete [15198.176969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15198.179344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15198.193710] Lustre: Skipped 1 previous similar message [15198.205629] Lustre: Skipped 1 previous similar message [15201.123853] LNetError: 465051:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.130 [15201.136919] LNetError: 465051:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [15201.151974] LNetError: Protocol error connecting to 192.168.202.130@tcp108 at host 192.168.202.130:988: is it running a compatible version of Lustre? [15201.173981] LNetError: Skipped 20 previous similar messages [15203.298640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15203.308802] Lustre: Skipped 1 previous similar message [15203.890641] Lustre: server umount lustre-MDT0000 complete [15208.002292] LustreError: 465034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790786938 with bad export cookie 7453445546598872404 [15208.027046] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15208.475120] Lustre: server umount lustre-MDT0001 complete [15219.546077] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [15222.567569] Key type lgssc unregistered [15223.072782] LNet: 468930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15223.081836] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15223.084647] LNet: Removed LNI 192.168.202.130@tcp [15224.098032] LNet: Removed LNI 192.168.202.130@tcp100 [15224.103296] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15224.116489] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15224.123328] LNet: Removed LNI 192.168.202.130@tcp101 [15225.187356] LNet: Removed LNI 192.168.202.130@tcp102 [15225.202381] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15225.224057] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15225.245435] LNet: Removed LNI 192.168.202.130@tcp103 [15226.271989] LNet: Removed LNI 192.168.202.130@tcp104 [15227.295187] LNet: Removed LNI 192.168.202.130@tcp105 [15227.301592] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15227.310761] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15228.322327] LNet: Removed LNI 192.168.202.130@tcp106 [15229.343126] LNet: Removed LNI 192.168.202.130@tcp107 [15230.368442] LNet: Removed LNI 192.168.202.130@tcp108 [15231.392949] LNet: Removed LNI 192.168.202.130@tcp109 [15231.398560] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15231.413033] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [15231.422115] LNet: Removed LNI 192.168.202.130@tcp110 [15232.483154] LNet: Removed LNI 192.168.202.130@tcp111 [15233.511148] LNet: Removed LNI 192.168.202.130@tcp112 [15234.529228] LNet: Removed LNI 192.168.202.130@tcp113 [15235.551741] LNet: Removed LNI 192.168.202.130@tcp114 [15236.575218] LNet: Removed LNI 192.168.202.130@tcp115 [15237.602081] LNet: Removed LNI 192.168.202.130@tcp116 [15238.623330] LNet: Removed LNI 192.168.202.130@tcp117 [15239.647584] LNet: Removed LNI 192.168.202.130@tcp118 [15239.651349] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15239.660264] LNetError: 468930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15239.667450] LNet: Removed LNI 192.168.202.130@tcp119 [15240.735793] LNet: Removed LNI 192.168.202.130@tcp120 [15240.740521] LNet: Removed LNI 192.168.202.130@tcp121 [15241.759509] LNet: Removed LNI 192.168.202.130@tcp122 [15242.783156] LNet: Removed LNI 192.168.202.130@tcp123 [15243.807205] LNet: Removed LNI 192.168.202.130@tcp124 [15244.831636] LNet: Removed LNI 192.168.202.130@tcp125 [15245.856273] LNet: Removed LNI 192.168.202.130@tcp126 [15246.879160] LNet: Removed LNI 192.168.202.130@tcp127 [15247.903219] LNet: Removed LNI 192.168.202.130@tcp128 [15248.929060] LNet: Removed LNI 192.168.202.130@tcp129 [15248.930968] LNet: Removed LNI 192.168.202.130@tcp130 [15248.932580] LNet: Removed LNI 192.168.202.130@tcp131 [15248.935794] LNet: Removed LNI 192.168.202.130@tcp132 [15248.938767] LNet: Removed LNI 192.168.202.130@tcp133 [15248.941381] LNet: Removed LNI 192.168.202.130@tcp134 [15248.950901] LNet: Removed LNI 192.168.202.130@tcp135 [15249.633880] Key type .llcrypt unregistered [15249.638169] Key type ._llcrypt unregistered [15271.584958] Key type ._llcrypt registered [15271.588115] Key type .llcrypt registered [15271.668871] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [15291.351231] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [15293.029635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15293.065332] alg: No test for adler32 (adler32-zlib) [15294.353891] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [15294.929288] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [15296.728097] Key type lgssc registered [15298.308953] Lustre: Echo OBD driver; http://www.lustre.org/ [15340.086632] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:51:09 (1790787069) [15359.850875] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [15363.457423] Key type lgssc unregistered [15363.878552] LNet: 473033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15363.897418] LNetError: 473033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15363.933666] LNet: Removed LNI 192.168.202.130@tcp [15365.230530] Key type .llcrypt unregistered [15365.236127] Key type ._llcrypt unregistered [15385.366914] Key type ._llcrypt registered [15385.369266] Key type .llcrypt registered [15385.477703] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [15386.720237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15386.743577] alg: No test for adler32 (adler32-zlib) [15387.813695] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [15388.026326] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [15389.687289] Key type lgssc registered [15390.660744] Lustre: Echo OBD driver; http://www.lustre.org/ [15400.627245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [15400.663611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15401.977380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15402.048480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15402.183986] Lustre: lustre-MDT0000: new disk, initializing [15402.269577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15402.290579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15406.297732] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15416.408196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15416.601265] Lustre: 474266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15416.708222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15416.724290] Lustre: Skipped 1 previous similar message [15416.802435] Lustre: lustre-MDT0001: new disk, initializing [15416.947687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15417.011680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15417.026844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15422.113788] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15431.655306] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15438.429397] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15445.958440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15446.273960] Lustre: lustre-OST0000: new disk, initializing [15446.277656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15446.284599] Lustre: 476233:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15446.338654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15452.461818] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15454.278817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15454.298245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15454.442403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15462.616549] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15467.966573] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing lnet_if_list [15473.725497] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15473.750685] LNet: Added LNI 192.168.202.130@tcp10 [8/256/0/180] [15479.389380] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15479.430779] LNet: Added LNI 192.168.202.130@tcp11 [8/256/0/180] [15485.715836] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15485.779265] LNet: Added LNI 192.168.202.130@tcp12 [8/256/0/180] [15490.892223] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15490.919352] LNet: Added LNI 192.168.202.130@tcp13 [8/256/0/180] [15495.698535] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15495.728418] LNet: Added LNI 192.168.202.130@tcp14 [8/256/0/180] [15502.046522] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15502.070438] LNet: Added LNI 192.168.202.130@tcp15 [8/256/0/180] [15507.010728] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15507.043584] LNet: Added LNI 192.168.202.130@tcp16 [8/256/0/180] [15511.693464] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15511.732844] LNet: Added LNI 192.168.202.130@tcp17 [8/256/0/180] [15517.861155] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15517.909874] LNet: Added LNI 192.168.202.130@tcp18 [8/256/0/180] [15522.935370] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15522.980218] LNet: Added LNI 192.168.202.130@tcp19 [8/256/0/180] [15528.793836] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15528.852578] LNet: Added LNI 192.168.202.130@tcp20 [8/256/0/180] [15534.560781] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15534.587758] LNet: Added LNI 192.168.202.130@tcp21 [8/256/0/180] [15539.682395] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15539.706299] LNet: Added LNI 192.168.202.130@tcp22 [8/256/0/180] [15544.414778] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15544.442508] LNet: Added LNI 192.168.202.130@tcp23 [8/256/0/180] [15549.516957] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15549.570190] LNet: Added LNI 192.168.202.130@tcp24 [8/256/0/180] [15556.025488] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15556.087093] LNet: Added LNI 192.168.202.130@tcp25 [8/256/0/180] [15564.408844] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15564.434559] LNet: Added LNI 192.168.202.130@tcp26 [8/256/0/180] [15571.888822] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15571.946061] LNet: Added LNI 192.168.202.130@tcp27 [8/256/0/180] [15579.245780] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15579.291170] LNet: Added LNI 192.168.202.130@tcp28 [8/256/0/180] [15585.764358] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15585.833933] LNet: Added LNI 192.168.202.130@tcp29 [8/256/0/180] [15593.805943] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15593.836251] LNet: Added LNI 192.168.202.130@tcp30 [8/256/0/180] [15601.157544] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15601.186765] LNet: Added LNI 192.168.202.130@tcp31 [8/256/0/180] [15608.874821] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15608.918565] LNet: Added LNI 192.168.202.130@tcp32 [8/256/0/180] [15618.683538] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15618.726082] LNet: Added LNI 192.168.202.130@tcp33 [8/256/0/180] [15626.960484] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15627.015027] LNet: Added LNI 192.168.202.130@tcp34 [8/256/0/180] [15635.450333] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15635.468931] LNet: Added LNI 192.168.202.130@tcp35 [8/256/0/180] [15643.422698] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15643.453890] LNet: Added LNI 192.168.202.130@tcp36 [8/256/0/180] [15650.538279] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15650.583710] LNet: Added LNI 192.168.202.130@tcp37 [8/256/0/180] [15657.652193] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15657.684796] LNet: Added LNI 192.168.202.130@tcp38 [8/256/0/180] [15665.393803] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15665.464816] LNet: Added LNI 192.168.202.130@tcp39 [8/256/0/180] [15674.110720] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15674.150761] LNet: Added LNI 192.168.202.130@tcp40 [8/256/0/180] [15682.779584] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15682.802915] LNet: Added LNI 192.168.202.130@tcp41 [8/256/0/180] [15691.155424] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15691.173123] LNet: Added LNI 192.168.202.130@tcp42 [8/256/0/180] [15699.651541] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15699.671602] LNet: Added LNI 192.168.202.130@tcp43 [8/256/0/180] [15707.264850] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15707.314987] LNet: Added LNI 192.168.202.130@tcp44 [8/256/0/180] [15716.662310] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15716.715849] LNet: Added LNI 192.168.202.130@tcp45 [8/256/0/180] [15724.761240] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15724.831086] LNet: Added LNI 192.168.202.130@tcp46 [8/256/0/180] [15732.807502] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15732.828786] LNet: Added LNI 192.168.202.130@tcp47 [8/256/0/180] [15740.767391] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15740.820713] LNet: Added LNI 192.168.202.130@tcp48 [8/256/0/180] [15751.440237] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15751.479129] LNet: Added LNI 192.168.202.130@tcp49 [8/256/0/180] [15762.885534] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15762.990697] LNet: Added LNI 192.168.202.130@tcp50 [8/256/0/180] [15787.500981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15787.504298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15787.515292] Lustre: Skipped 2 previous similar messages [15787.529054] Lustre: Skipped 1 previous similar message [15792.609134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15792.611402] Lustre: Skipped 1 previous similar message [15793.146731] Lustre: server umount lustre-MDT0000 complete [15799.082527] LustreError: 474255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787530 with bad export cookie 3830095282289314367 [15799.093810] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15799.099771] LustreError: 474255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15799.710223] Lustre: server umount lustre-MDT0001 complete [15818.214461] Lustre: 473528:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790787533/real 1790787533] req@000000008511ca6c x1877776393227648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790787549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15818.261988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15820.319592] Lustre: 473528:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790787535/real 1790787535] req@000000005aed7d46 x1877776393227776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790787551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15822.441842] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [15823.328450] Lustre: 473526:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790787538/real 1790787538] req@00000000535f8fad x1877776393227904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790787554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15825.375371] Lustre: 473527:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790787540/real 1790787540] req@00000000bf282d4e x1877776393228160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790787556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15831.532488] Lustre: 473529:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790787546/real 1790787546] req@00000000dc58a512 x1877776393228288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790787562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15831.677681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15844.967596] LustreError: 473525:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c0ecff58 x1877776393229056/t0(0) o250->MGC192.168.202.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15845.291450] LustreError: 476226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15845.303553] LustreError: 476226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15845.374591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15848.420516] LustreError: 476227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15850.445218] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15852.648332] LustreError: 476226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15852.653396] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15858.148783] LustreError: 476227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15860.873949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15861.393385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15863.482237] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15866.221274] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [15874.941978] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15881.484438] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15904.779087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15904.811098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15904.818649] Lustre: Skipped 1 previous similar message [15909.661901] Lustre: server umount lustre-OST0000 complete [15914.979774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15914.999987] Lustre: Skipped 1 previous similar message [15915.013422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15915.021162] Lustre: Skipped 1 previous similar message [15920.099680] LustreError: 500484:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15920.112398] LustreError: 500484:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15921.867506] Lustre: server umount lustre-MDT0000 complete [15925.371207] LustreError: 474257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787656 with bad export cookie 3830095282289317041 [15925.376500] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15925.391471] LustreError: 474257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15925.721689] Lustre: server umount lustre-MDT0001 complete [15938.817973] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [15941.534619] Key type lgssc unregistered [15941.867663] LNet: 503294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15941.873949] LNetError: 503294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15942.879185] LNet: Removed LNI 192.168.202.130@tcp [15942.881138] LNetError: 503294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15942.893394] LNet: Removed LNI 192.168.202.130@tcp10 [15942.899191] LNet: Removed LNI 192.168.202.130@tcp11 [15942.903280] LNet: Removed LNI 192.168.202.130@tcp12 [15942.908309] LNet: Removed LNI 192.168.202.130@tcp13 [15942.912084] LNet: Removed LNI 192.168.202.130@tcp14 [15942.915047] LNet: Removed LNI 192.168.202.130@tcp15 [15942.919417] LNet: Removed LNI 192.168.202.130@tcp16 [15942.925393] LNet: Removed LNI 192.168.202.130@tcp17 [15942.928533] LNet: Removed LNI 192.168.202.130@tcp18 [15942.931082] LNet: Removed LNI 192.168.202.130@tcp19 [15942.933572] LNet: Removed LNI 192.168.202.130@tcp20 [15942.936561] LNet: Removed LNI 192.168.202.130@tcp21 [15942.938301] LNet: Removed LNI 192.168.202.130@tcp22 [15942.947427] LNet: Removed LNI 192.168.202.130@tcp23 [15942.950249] LNet: Removed LNI 192.168.202.130@tcp24 [15942.952912] LNet: Removed LNI 192.168.202.130@tcp25 [15942.955427] LNet: Removed LNI 192.168.202.130@tcp26 [15942.958190] LNet: Removed LNI 192.168.202.130@tcp27 [15942.959924] LNet: Removed LNI 192.168.202.130@tcp28 [15942.962968] LNet: Removed LNI 192.168.202.130@tcp29 [15942.965205] LNet: Removed LNI 192.168.202.130@tcp30 [15942.969223] LNet: Removed LNI 192.168.202.130@tcp31 [15942.972156] LNet: Removed LNI 192.168.202.130@tcp32 [15942.975368] LNet: Removed LNI 192.168.202.130@tcp33 [15942.978977] LNet: Removed LNI 192.168.202.130@tcp34 [15942.981242] LNet: Removed LNI 192.168.202.130@tcp35 [15942.984652] LNet: Removed LNI 192.168.202.130@tcp36 [15942.988949] LNet: Removed LNI 192.168.202.130@tcp37 [15942.991427] LNet: Removed LNI 192.168.202.130@tcp38 [15943.002299] LNet: Removed LNI 192.168.202.130@tcp39 [15943.005563] LNet: Removed LNI 192.168.202.130@tcp40 [15943.009249] LNet: Removed LNI 192.168.202.130@tcp41 [15943.014948] LNet: Removed LNI 192.168.202.130@tcp42 [15943.017874] LNet: Removed LNI 192.168.202.130@tcp43 [15943.022913] LNet: Removed LNI 192.168.202.130@tcp44 [15943.024820] LNet: Removed LNI 192.168.202.130@tcp45 [15943.027501] LNet: Removed LNI 192.168.202.130@tcp46 [15943.031417] LNet: Removed LNI 192.168.202.130@tcp47 [15943.034280] LNet: Removed LNI 192.168.202.130@tcp48 [15943.037170] LNet: Removed LNI 192.168.202.130@tcp49 [15943.052897] LNet: Removed LNI 192.168.202.130@tcp50 [15943.855108] Key type .llcrypt unregistered [15943.857934] Key type ._llcrypt unregistered [15967.080184] Key type ._llcrypt registered [15967.083283] Key type .llcrypt registered [15967.222961] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [15987.771242] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [15988.897874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15989.047184] alg: No test for adler32 (adler32-zlib) [15990.069218] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [15990.395959] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [15992.127211] Key type lgssc registered [15993.900299] Lustre: Echo OBD driver; http://www.lustre.org/ [16037.404645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16037.429908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16038.906426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16038.964816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16039.093571] Lustre: lustre-MDT0000: new disk, initializing [16039.199142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16039.226400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16044.600745] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16058.247870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16058.385992] Lustre: 506508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16058.446571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16058.454152] Lustre: Skipped 1 previous similar message [16058.604342] Lustre: lustre-MDT0001: new disk, initializing [16058.755428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16058.839792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16058.869427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16064.522594] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16076.902488] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16083.289667] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16094.446751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16094.836742] Lustre: *** cfs_fail_loc=90f, val=0*** [16094.949919] Lustre: lustre-OST0000: new disk, initializing [16094.952607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16094.961952] Lustre: 508524:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16095.101792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16101.944215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16101.954851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16102.117943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16102.994432] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16114.504553] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16127.986354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16127.989477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16127.994617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16128.004927] LustreError: Skipped 1 previous similar message [16128.026674] Lustre: Skipped 1 previous similar message [16129.839891] Lustre: server umount lustre-OST0000 complete [16135.651249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16135.663172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16135.686417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16135.699745] Lustre: Skipped 1 previous similar message [16138.213151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16138.214942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16140.426384] Lustre: server umount lustre-MDT0000 complete [16145.739126] LustreError: 506500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790787876 with bad export cookie 15266123613296067522 [16145.764939] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16146.249729] Lustre: server umount lustre-MDT0001 complete [16162.115469] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [16165.639982] Key type lgssc unregistered [16165.948683] LNet: 510431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16165.969626] LNetError: 510431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16165.985719] LNet: Removed LNI 192.168.202.130@tcp [16167.142116] Key type .llcrypt unregistered [16167.146793] Key type ._llcrypt unregistered [16192.129754] Key type ._llcrypt registered [16192.131438] Key type .llcrypt registered [16192.203801] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [16193.550328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16193.573213] alg: No test for adler32 (adler32-zlib) [16194.695818] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [16194.898390] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [16196.615206] Key type lgssc registered [16197.722504] Lustre: Echo OBD driver; http://www.lustre.org/ [16207.662800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16207.691047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16209.143984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16214.766562] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16219.624685] LustreError: 511718:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16219.655943] LustreError: 511718:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16224.745310] LustreError: 511717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16226.697787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16227.117262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16232.857832] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16241.828427] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16248.367272] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16257.270171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16257.767151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16266.114845] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16278.404680] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16292.215695] loop0: detected capacity change from 0 to 204800000 [16292.504127] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16292.526882] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16292.545114] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16300.463813] loop0: detected capacity change from 0 to 204800000 [16300.861180] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16300.870185] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16300.879497] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16310.778450] loop0: detected capacity change from 0 to 204800000 [16319.156582] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16319.620094] Lustre: fs15246-OST0000: new disk, initializing [16319.634598] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16319.650508] Lustre: 515315:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16319.786179] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16328.961246] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16340.531469] loop1: detected capacity change from 0 to 204800000 [16348.022448] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16348.192223] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16348.271904] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16348.338064] Lustre: fs15246-MDT0000: new disk, initializing [16348.577837] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16348.609761] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16348.767689] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16349.856252] Lustre: 516439:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000053bb4608 x1877777239300864/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790788096 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16354.155097] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16358.955237] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16358.969923] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16359.037145] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16386.534824] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16386.580744] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16391.672459] LustreError: 511712:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16391.756907] Lustre: server umount fs15246-MDT0000 complete [16396.777564] LustreError: 511713:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16401.911864] LustreError: 511712:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16404.465541] Lustre: server umount fs15246-OST0000 complete [16420.837825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16420.841273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16420.873949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16422.374908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16422.390662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16422.404881] Lustre: Skipped 1 previous similar message [16424.504292] Lustre: server umount lustre-OST0000 complete [16432.096544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16432.108189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16432.124309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16437.728313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16437.741165] Lustre: Skipped 2 previous similar messages [16443.359123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16443.597833] Lustre: server umount lustre-MDT0000 complete [16447.776967] LustreError: 511697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788178 with bad export cookie 9406273963616241548 [16447.776967] LustreError: 511698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788178 with bad export cookie 9406273963616241548 [16447.777957] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16447.790378] LustreError: 511697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16448.422550] Lustre: server umount lustre-MDT0001 complete [16460.986391] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [16463.797496] Key type lgssc unregistered [16464.126550] LNet: 519537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16464.135956] LNetError: 519537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16464.166214] LNet: Removed LNI 192.168.202.130@tcp [16465.085570] Key type .llcrypt unregistered [16465.088193] Key type ._llcrypt unregistered [16488.469872] Key type ._llcrypt registered [16488.472723] Key type .llcrypt registered [16488.620943] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [16490.241301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16490.284902] alg: No test for adler32 (adler32-zlib) [16491.314343] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [16491.633661] LNet: Added LNI 192.168.202.130@tcp [8/256/0/180] [16493.375890] Key type lgssc registered [16494.997325] Lustre: Echo OBD driver; http://www.lustre.org/ [16540.578280] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [16552.674473] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [16595.511426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [16595.598368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16597.014653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16597.098114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16597.238607] Lustre: lustre-MDT0000: new disk, initializing [16597.441818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16597.478831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16604.341432] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16620.586542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16620.700207] Lustre: 524024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16620.761148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16620.765373] Lustre: Skipped 1 previous similar message [16620.852789] Lustre: lustre-MDT0001: new disk, initializing [16620.958966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16621.001343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16621.017453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16627.933519] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16643.789795] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16651.559739] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16659.684234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16660.045677] Lustre: lustre-OST0000: new disk, initializing [16660.052061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16660.065308] Lustre: 525996:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16660.215181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16667.314425] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_default_debug -1 all [16668.259353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16668.292855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16668.453916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16678.762704] Lustre: DEBUG MARKER: oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16685.013240] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16685.299944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16690.631340] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16690.932649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16694.243412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16694.248242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16694.288741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16694.308106] Lustre: Skipped 1 previous similar message [16699.372636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16699.391577] Lustre: Skipped 1 previous similar message [16700.044032] Lustre: server umount lustre-OST0000 complete [16708.066368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16708.077239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16708.096367] Lustre: Skipped 1 previous similar message [16708.111756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16709.601415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16714.736519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16714.739076] Lustre: Skipped 2 previous similar messages [16719.327138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16719.671902] Lustre: server umount lustre-MDT0000 complete [16723.700046] LustreError: 524015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790788454 with bad export cookie 3968003791598028445 [16723.708888] LustreError: MGC192.168.202.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16723.711165] LustreError: 524015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16724.211026] Lustre: server umount lustre-MDT0001 complete [16756.261828] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing set_hostid [16768.606855] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing load_modules_local [16810.931395] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16524 sec ========= 13:15:39 (1790788539) [16813.736583] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:15:42 (1790788542) === [16818.783252] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:15:47 (1790788547) === [16866.318890] Lustre: DEBUG MARKER: oleg230-server.virtnet: executing unload_modules_local [16869.515410] Key type lgssc unregistered [16869.805540] LNet: 532648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16869.814304] LNetError: 532648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16870.887968] LNet: Removed LNI 192.168.202.130@tcp [16872.302869] Key type .llcrypt unregistered [16872.314688] Key type ._llcrypt unregistered