[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1657972521 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002530] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011188] LSM: Security Framework initializing [ 0.012085] Yama: becoming mindful. [ 0.013115] SELinux: Initializing. [ 0.014117] *** VALIDATE selinux *** [ 0.025087] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.036000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.036393] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037319] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039125] *** VALIDATE tmpfs *** [ 0.041347] *** VALIDATE proc *** [ 0.043238] *** VALIDATE cgroup *** [ 0.045015] *** VALIDATE cgroup2 *** [ 0.046810] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.048190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.049019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.050055] Spectre V2 : User space: Vulnerable [ 0.051022] Speculative Store Bypass: Vulnerable [ 0.056005] debug: unmapping init [mem 0xffffffff87059000-0xffffffff87060fff] [ 0.061000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.062220] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.063045] ... version: 2 [ 0.064033] ... bit width: 48 [ 0.066033] ... generic registers: 4 [ 0.067558] ... value mask: 0000ffffffffffff [ 0.068037] ... max period: 00007fffffffffff [ 0.069035] ... fixed-purpose events: 3 [ 0.070084] ... event mask: 000000070000000f [ 0.071802] rcu: Hierarchical SRCU implementation. [ 0.077631] smp: Bringing up secondary CPUs ... [ 0.078964] x86: Booting SMP configuration: [ 0.079073] .... node #0, CPUs: #1 #2 #3 [ 0.108022] smp: Brought up 1 node, 4 CPUs [ 0.110023] smpboot: Max logical packages: 1 [ 0.111025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.171170] node 0 deferred pages initialised in 58ms [ 0.180016] devtmpfs: initialized [ 0.181350] x86/mm: Memory block size: 128MB [ 0.186719] gcov: version magic: 0x41383552 [ 0.190171] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.192107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.193410] pinctrl core: initialized pinctrl subsystem [ 0.194386] [ 0.195022] ************************************************************* [ 0.196033] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197038] ** ** [ 0.198033] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199038] ** ** [ 0.200023] ** This means that this kernel is built to expose internal ** [ 0.201030] ** IOMMU data structures, which may compromise security on ** [ 0.202021] ** your system. ** [ 0.203039] ** ** [ 0.204025] ** If you see this message and you are not debugging the ** [ 0.205022] ** kernel, report this immediately to your vendor! ** [ 0.206032] ** ** [ 0.207025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208025] ************************************************************* [ 0.210695] NET: Registered protocol family 16 [ 0.213147] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.214095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.215120] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218015] cpuidle: using governor menu [ 0.223416] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.233227] PCI: Using configuration type 1 for base access [ 0.238169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.252000] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.254034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.263213] cryptd: max_cpu_qlen set to 1000 [ 0.270419] ACPI: Added _OSI(Module Device) [ 0.272026] ACPI: Added _OSI(Processor Device) [ 0.277026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.284022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.300546] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.318380] ACPI: Interpreter enabled [ 0.324068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.331020] ACPI: Using IOAPIC for interrupt routing [ 0.335658] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.355000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.392167] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.396066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.403027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.409107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.425250] acpiphp: Slot [2] registered [ 0.431186] acpiphp: Slot [5] registered [ 0.434592] acpiphp: Slot [6] registered [ 0.440000] acpiphp: Slot [7] registered [ 0.446068] acpiphp: Slot [8] registered [ 0.449662] acpiphp: Slot [9] registered [ 0.452226] acpiphp: Slot [10] registered [ 0.455692] acpiphp: Slot [3] registered [ 0.458201] acpiphp: Slot [4] registered [ 0.460140] acpiphp: Slot [11] registered [ 0.462137] acpiphp: Slot [12] registered [ 0.464142] acpiphp: Slot [13] registered [ 0.467417] acpiphp: Slot [14] registered [ 0.469144] acpiphp: Slot [15] registered [ 0.471523] acpiphp: Slot [16] registered [ 0.474214] acpiphp: Slot [17] registered [ 0.476338] acpiphp: Slot [18] registered [ 0.478151] acpiphp: Slot [19] registered [ 0.480433] acpiphp: Slot [20] registered [ 0.484389] acpiphp: Slot [21] registered [ 0.487189] acpiphp: Slot [22] registered [ 0.490356] acpiphp: Slot [23] registered [ 0.494043] acpiphp: Slot [24] registered [ 0.497179] acpiphp: Slot [25] registered [ 0.500179] acpiphp: Slot [26] registered [ 0.503984] acpiphp: Slot [27] registered [ 0.508935] acpiphp: Slot [28] registered [ 0.511496] acpiphp: Slot [29] registered [ 0.514899] acpiphp: Slot [30] registered [ 0.517172] acpiphp: Slot [31] registered [ 0.520099] PCI host bridge to bus 0000:00 [ 0.523033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.528034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.533033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.539034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.544949] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.550038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.555083] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.563245] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.572669] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.606022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.613340] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.623025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.629025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.641034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.654899] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.666094] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.679023] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.687059] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 20507 usecs [ 0.697468] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.710022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.751022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.765023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.776533] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.791373] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.813024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.892000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.922114] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.943027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.962024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.989027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.013000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.023020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.036028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.063023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.076457] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.082000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.089025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.115025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.134030] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.149023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.162036] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.209021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.239065] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.255028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.273024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.310025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.341391] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.348398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.354301] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.359000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.367439] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.380070] iommu: Default domain type: Passthrough [ 1.384543] SCSI subsystem initialized [ 1.389242] ACPI: bus type USB registered [ 1.393332] usbcore: registered new interface driver usbfs [ 1.397865] usbcore: registered new interface driver hub [ 1.402656] usbcore: registered new device driver usb [ 1.407259] pps_core: LinuxPPS API ver. 1 registered [ 1.409016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.418097] PTP clock support registered [ 1.425000] EDAC MC: Ver: 3.0.0 [ 1.426251] PCI: Using ACPI for IRQ routing [ 1.431272] NetLabel: Initializing [ 1.432016] NetLabel: domain hash size = 128 [ 1.436020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.441112] NetLabel: unlabeled traffic allowed by default [ 1.448053] vgaarb: loaded [ 1.454746] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.458019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.471623] clocksource: Switched to clocksource kvm-clock [ 1.740320] VFS: Disk quotas dquot_6.6.0 [ 1.743845] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.750219] *** VALIDATE ramfs *** [ 1.754262] *** VALIDATE hugetlbfs *** [ 1.759154] pnp: PnP ACPI init [ 1.766147] pnp: PnP ACPI: found 6 devices [ 1.809734] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.819704] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.825286] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.831073] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.837759] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.844586] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.849310] NET: Registered protocol family 2 [ 1.854671] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.865100] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.872420] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.880824] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.889483] TCP: Hash tables configured (established 65536 bind 65536) [ 1.900206] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.910187] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.919333] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.928966] NET: Registered protocol family 1 [ 1.936186] RPC: Registered named UNIX socket transport module. [ 1.943302] RPC: Registered udp transport module. [ 1.948052] RPC: Registered tcp transport module. [ 1.952979] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.960331] NET: Registered protocol family 44 [ 1.965090] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.970959] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.978134] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.983136] PCI: CLS 0 bytes, default 64 [ 1.988942] Unpacking initramfs... [ 8.510600] debug: unmapping init [mem 0xffff8f23fcc54000-0xffff8f23fffbffff] [ 8.526612] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 8.533995] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 8.542760] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 11.791928] Initialise system trusted keyrings [ 11.802494] Key type blacklist registered [ 11.814470] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 11.870523] zbud: loaded [ 11.900544] *** VALIDATE nfs *** [ 11.901822] *** VALIDATE nfs4 *** [ 11.911922] pstore: using deflate compression [ 11.956762] Platform Keyring initialized [ 12.949066] NET: Registered protocol family 38 [ 12.955037] Key type asymmetric registered [ 12.974487] Asymmetric key parser 'x509' registered [ 12.982839] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 13.002033] io scheduler mq-deadline registered [ 13.010324] io scheduler kyber registered [ 13.028023] io scheduler bfq registered [ 13.036429] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 13.048697] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 13.070441] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 13.086136] ACPI: Power Button [PWRF] [ 13.123256] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 13.168268] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 13.250296] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 13.277481] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 13.345843] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 13.416111] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 13.522915] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 13.578247] Non-volatile memory driver v1.3 [ 13.587222] Linux agpgart interface v0.103 [ 13.925159] virtio_blk virtio1: [vda] 149840 512-byte logical blocks (76.7 MB/73.2 MiB) [ 13.953836] vda: detected capacity change from 0 to 76718080 [ 14.101205] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 14.119443] vdb: detected capacity change from 0 to 1073741824 [ 14.210447] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 14.230236] vdc: detected capacity change from 0 to 2621440000 [ 14.385492] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 14.412927] vdd: detected capacity change from 0 to 2621440000 [ 14.549722] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.559421] vde: detected capacity change from 0 to 4294967296 [ 14.644925] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.649562] vdf: detected capacity change from 0 to 4294967296 [ 14.669344] libphy: Fixed MDIO Bus: probed [ 14.706707] usbcore: registered new interface driver usbserial_generic [ 14.713056] usbserial: USB Serial support registered for generic [ 14.723403] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 14.740777] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 14.748394] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 14.761737] mousedev: PS/2 mouse device common for all mice [ 14.780871] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 14.781487] rtc_cmos 00:05: RTC can wake from S4 [ 14.800164] rtc_cmos 00:05: registered as rtc0 [ 14.816802] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 14.834875] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 14.844561] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 14.855176] intel_pstate: CPU model not supported [ 14.870030] hid: raw HID events driver (C) Jiri Kosina [ 14.937607] usbcore: registered new interface driver usbhid [ 14.955454] usbhid: USB HID core driver [ 14.961256] drop_monitor: Initializing network drop monitor service [ 14.970493] Initializing XFRM netlink socket [ 14.980569] NET: Registered protocol family 10 [ 15.004464] Segment Routing with IPv6 [ 15.024832] NET: Registered protocol family 17 [ 15.046867] mpls_gso: MPLS GSO support [ 15.108626] RAS: Correctable Errors collector initialized. [ 15.129829] AVX version of gcm_enc/dec engaged. [ 15.139218] AES CTR mode by8 optimization enabled [ 15.794377] sched_clock: Marking stable (15794355264, 0)->(18628441024, -2834085760) [ 15.800971] registered taskstats version 1 [ 15.839399] Loading compiled-in X.509 certificates [ 15.841561] zswap: loaded using pool lzo/zbud [ 16.096081] Key type big_key registered [ 16.281456] Key type encrypted registered [ 16.285993] ima: No TPM chip found, activating TPM-bypass! [ 16.295063] ima: Allocated hash algorithm: sha1 [ 16.309435] ima: No architecture policies found [ 16.330183] evm: Initialising EVM extended attributes: [ 16.338574] evm: security.selinux [ 16.356632] evm: security.ima [ 16.371198] evm: security.capability [ 16.392295] evm: HMAC attrs: 0x1 [ 16.433666] rtc_cmos 00:05: setting system clock to 2026-10-02 17:45:51 UTC (1790963151) [ 16.472060] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 16.517894] debug: unmapping init [mem 0xffffffff86d82000-0xffffffff87058fff] [ 16.543749] Write protecting the kernel read-only data: 28672k [ 16.595118] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 16.612623] debug: unmapping init [mem 0xffffffff85d14000-0xffffffff85dfffff] [ 17.026203] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.106918] systemd[1]: Detected virtualization kvm. [ 17.127684] systemd[1]: Detected architecture x86-64. [ 17.139924] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 17.421141] systemd[1]: No hostname configured. [ 17.448211] systemd[1]: Set hostname to . [ 17.450351] random: systemd: uninitialized urandom read (16 bytes read) [ 17.493907] systemd[1]: Initializing machine ID from random generator. [ 18.087357] random: ln: uninitialized urandom read (6 bytes read) [ 18.875163] random: systemd: uninitialized urandom read (16 bytes read) [ 18.894534] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 18.930957] random: systemd: uninitialized urandom read (16 bytes read) [ 18.945614] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 18.980434] random: systemd: uninitialized urandom read (16 bytes read) [ 19.017397] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 22.328953] device-mapper: uevent: version 1.0.3 [ 22.331630] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 26.228193] virtio_net virtio0 ens2: renamed from eth0 [ 26.702894] scsi host0: ata_piix [ 26.740244] scsi host1: ata_piix [ 26.765395] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 26.797641] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 26.810134] random: fast init done [ 31.989552] random: crng init done [ 32.010177] random: 5 urandom warning(s) missed due to ratelimiting [ 36.253882] hrtimer: interrupt took 3623338 ns [ 36.262813] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 41.526094] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 50.283524] printk: systemd: 26 output lines suppressed due to ratelimiting [ 51.691913] SELinux: Disabled at runtime. [ 52.253573] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 52.321292] systemd[1]: Detected virtualization kvm. [ 52.330510] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 55.928618] systemd[1]: initrd-switch-root.service: Succeeded. [ 55.972351] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 55.997399] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 56.020566] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 56.036550] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 56.113817] systemd[1]: Starting Journal Service... Starting Journal Service... [ 56.159729] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ 56.701091] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 58.597223] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 60.494436] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 60.544978] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 61.357973] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 61.407762] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 69.655409] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit)[ 70.890153] NFS: Registering the id_resolver key type [ 70.911906] Key type id_resolver registered [ 70.923279] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg433-server login: [ 134.958519] libcfs: loading out-of-tree module taints kernel. [ 135.048508] Key type ._llcrypt registered [ 135.050184] Key type .llcrypt registered [ 135.199637] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 155.918757] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 157.686437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 157.698654] alg: No test for adler32 (adler32-zlib) [ 159.210224] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 160.099084] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 161.792742] Key type lgssc registered [ 163.834982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 180.152342] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 221.238958] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 234.376843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 234.405800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.744510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 235.781255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 235.877711] Lustre: lustre-MDT0000: new disk, initializing [ 235.966031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.999696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 240.684586] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.528413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 253.684285] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 253.737826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 253.741563] Lustre: Skipped 1 previous similar message [ 253.877112] Lustre: lustre-MDT0001: new disk, initializing [ 253.985765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 254.021166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 254.039826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 258.497863] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.912958] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 274.159504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.448806] Lustre: lustre-OST0000: new disk, initializing [ 274.456573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 274.467857] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 274.530238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 276.067245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 276.084663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 276.203406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 281.366716] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.078789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 296.239311] Lustre: lustre-OST0001: new disk, initializing [ 296.241935] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 296.246932] Lustre: 9533:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 296.318478] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 302.389655] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 305.207096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 305.215382] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 305.327806] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 315.093249] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 320.634741] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 327.798933] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing check_logdir /tmp/testlogs/ [ 334.090656] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing yml_node [ 340.523249] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 343.926503] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 347.908711] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 350.701850] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 13:51:23 EDT 2026 [ 373.359068] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 375.139698] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 392.174714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 392.177836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 392.178088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 392.187171] Lustre: Skipped 2 previous similar messages [ 392.207099] Lustre: Skipped 3 previous similar messages [ 397.011493] Lustre: server umount lustre-MDT0000 complete [ 400.723476] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963535 with bad export cookie 4323170764733631454 [ 400.725885] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 400.745893] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 402.406412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.406442] LustreError: 7752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.406796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 402.415930] Lustre: Skipped 2 previous similar messages [ 402.431894] LustreError: 7752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 405.416421] LustreError: 6532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.421109] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 405.431761] LustreError: 6532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 405.435467] Lustre: Skipped 2 previous similar messages [ 410.544612] LustreError: 7752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.550414] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.568513] LustreError: 7752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 410.586099] Lustre: Skipped 1 previous similar message [ 415.200770] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 415.385663] Lustre: server umount lustre-MDT0001 complete [ 420.118098] Lustre: server umount lustre-OST0000 complete [ 423.919592] Lustre: server umount lustre-OST0001 complete [ 430.711466] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 438.917395] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 470.069366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.279364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 470.311592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 470.399478] Lustre: lustre-MDT0000: new disk, initializing [ 470.457895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.472419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 473.968670] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 482.539243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.632199] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 482.676698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 482.679183] Lustre: Skipped 1 previous similar message [ 482.733455] Lustre: lustre-MDT0001: new disk, initializing [ 482.806693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.844276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 482.856975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 485.612812] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 492.135504] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.994603] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 505.407653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.783973] Lustre: lustre-OST0000: new disk, initializing [ 505.789793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 505.805389] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 506.029084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.486452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 507.505737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.678739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 516.503062] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 533.260621] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 543.080821] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 543.622608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 551.114871] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 551.299887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 553.953577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.961726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.979317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.989959] Lustre: Skipped 1 previous similar message [ 559.693387] Lustre: server umount lustre-OST0000 complete [ 564.708693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 564.716540] LustreError: Skipped 1 previous similar message [ 564.721839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.736590] Lustre: Skipped 1 previous similar message [ 564.743336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.747976] Lustre: Skipped 2 previous similar messages [ 570.731313] Lustre: server umount lustre-MDT0000 complete [ 572.399429] LustreError: 16313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.426281] LustreError: 16313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 574.643193] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963709 with bad export cookie 4323170764733632987 [ 574.656934] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.263438] Lustre: server umount lustre-MDT0001 complete [ 579.580237] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:55:13 (1790963713) [ 582.272619] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 584.226103] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:55:17 (1790963717) [ 590.698354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.277028] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.419742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.694962] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 601.572862] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.583674] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 604.766443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.130563] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 614.123310] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 618.539411] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 623.921425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 629.477860] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 635.184098] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 650.724358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 650.737167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.778331] Lustre: Skipped 1 previous similar message [ 650.806149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 650.818174] Lustre: Skipped 2 previous similar messages [ 651.578175] Lustre: server umount lustre-OST0000 complete [ 661.475913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 661.515091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.385391] Lustre: server umount lustre-MDT0000 complete [ 671.201756] LustreError: 20198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.219972] LustreError: 20198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 672.519223] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963807 with bad export cookie 4323170764733633869 [ 672.522766] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.532226] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 672.996947] Lustre: server umount lustre-MDT0001 complete [ 690.509886] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 695.272792] Key type lgssc unregistered [ 695.899349] LNet: 23021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.920728] LNetError: 23021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.987397] LNet: Removed LNI 192.168.204.133@tcp [ 697.421155] Key type .llcrypt unregistered [ 697.423597] Key type ._llcrypt unregistered [ 726.464200] Key type ._llcrypt registered [ 726.468200] Key type .llcrypt registered [ 726.671878] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 728.117027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 728.380602] alg: No test for adler32 (adler32-zlib) [ 729.596494] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 729.855244] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 731.504219] Key type lgssc registered [ 732.750372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 741.601600] loop: module loaded [ 741.670647] loop0: detected capacity change from 0 to 204800000 [ 741.971735] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 741.977959] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 742.302841] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.079360] loop0: detected capacity change from 0 to 204800000 [ 749.282815] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.291852] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.303790] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 754.642412] loop0: detected capacity change from 0 to 204800000 [ 760.656343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 760.683923] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.935228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 761.960352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 762.055222] Lustre: lustre-MDT0000: new disk, initializing [ 762.130907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.151931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 767.854478] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 777.195985] loop1: detected capacity change from 0 to 204800000 [ 784.769257] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 785.338145] Lustre: lustre-OST07c6: new disk, initializing [ 785.355249] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 785.361744] Lustre: Skipped 1 previous similar message [ 785.384921] Lustre: 26204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 785.557395] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 793.366425] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 795.238424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 795.247665] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 795.324416] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 798.140457] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 798.146696] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 798.151242] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 798.158123] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 798.163676] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 798.168509] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 815.653525] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 815.666821] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 815.673536] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 815.679984] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 815.688934] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 815.696979] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 820.923094] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 820.933786] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 820.938923] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 820.947965] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.958920] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 820.983946] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 820.996230] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 821.016075] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.037174] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 821.058736] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.077815] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.088714] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.335806] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 824.348987] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 824.374796] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 824.393260] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.410034] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 824.422485] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.432336] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 824.446638] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.461128] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 824.479512] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 824.502160] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 824.515256] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.693173] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 832.710863] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 832.727675] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 832.741421] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 832.755588] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.769304] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 832.780053] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 832.799308] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 832.814536] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 832.831020] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 832.847807] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.861190] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 841.902875] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 841.931748] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 841.949577] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 841.965555] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 841.993363] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 842.012965] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.030850] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 842.043186] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.074350] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 842.093690] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 842.116028] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 842.127423] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 897.507601] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 897.508601] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.567582] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 899.731311] Lustre: server umount lustre-OST07c6 complete [ 909.963963] Lustre: server umount lustre-MDT0000 complete [ 928.855850] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 932.324662] Key type lgssc unregistered [ 932.833602] LNet: 28775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.844458] LNetError: 28775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.866691] LNet: Removed LNI 192.168.204.133@tcp [ 933.764621] Key type .llcrypt unregistered [ 933.768725] Key type ._llcrypt unregistered [ 955.328773] Key type ._llcrypt registered [ 955.331103] Key type .llcrypt registered [ 955.424845] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 956.666567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.742093] alg: No test for adler32 (adler32-zlib) [ 957.880637] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 958.203123] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 959.896246] Key type lgssc registered [ 961.671525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 969.546592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 969.581799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.080191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.867767] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 981.476611] LustreError: 30043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.507901] LustreError: 30043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 984.973026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.415520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 990.077794] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 997.959661] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.686755] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.329665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.772423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.898820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1019.626120] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1027.133604] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1047.522212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1047.544516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.567056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1049.057702] Lustre: server umount lustre-OST0000 complete [ 1055.716144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.734155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.668930] Lustre: server umount lustre-MDT0000 complete [ 1060.837464] LustreError: 30772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.867534] LustreError: 30772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1064.694732] LustreError: 30025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964199 with bad export cookie 16402695501296161134 [ 1064.704863] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.712440] LustreError: 30025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1074.373466] Lustre: server umount lustre-MDT0001 complete [ 1086.106974] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1088.806779] Key type lgssc unregistered [ 1089.200056] LNet: 33782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.206588] LNetError: 33782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.241483] LNet: Removed LNI 192.168.204.133@tcp [ 1090.488344] Key type .llcrypt unregistered [ 1090.490095] Key type ._llcrypt unregistered [ 1109.998290] Key type ._llcrypt registered [ 1110.001925] Key type .llcrypt registered [ 1110.103606] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1111.065557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.134659] alg: No test for adler32 (adler32-zlib) [ 1112.146264] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1112.377729] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1114.144245] Key type lgssc registered [ 1115.558538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1123.943738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1123.968678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.638456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.062819] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1135.075888] LustreError: 35048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.105044] LustreError: 35048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1140.194956] LustreError: 35049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.836069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1142.199229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.589927] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1153.653399] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1159.356526] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1165.961690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1166.381944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.494849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1173.054669] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1180.006977] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1190.882294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.883967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1190.922477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1190.944994] Lustre: Skipped 1 previous similar message [ 1196.012278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.040216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1200.054518] Lustre: Skipped 2 previous similar messages [ 1203.168108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1203.315584] Lustre: server umount lustre-OST0000 complete [ 1208.801814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1208.825101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.866690] Lustre: Skipped 1 previous similar message [ 1208.874089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.337676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.352987] Lustre: server umount lustre-MDT0000 complete [ 1215.459996] LustreError: 35049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.471423] LustreError: 35049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1216.775952] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964351 with bad export cookie 621212762803814720 [ 1216.778852] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1216.797773] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1217.043290] Lustre: server umount lustre-MDT0001 complete [ 1226.984858] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1229.675695] Key type lgssc unregistered [ 1229.938805] LNet: 38583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.944232] LNetError: 38583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.972344] LNet: Removed LNI 192.168.204.133@tcp [ 1230.717680] Key type .llcrypt unregistered [ 1230.719478] Key type ._llcrypt unregistered [ 1250.695217] Key type ._llcrypt registered [ 1250.697185] Key type .llcrypt registered [ 1250.804546] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1251.859835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.898569] alg: No test for adler32 (adler32-zlib) [ 1252.934377] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1253.160897] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1254.824278] Key type lgssc registered [ 1255.942729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1262.725562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1262.762412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.193730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.279272] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1274.348682] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.376630] LustreError: 39975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1275.764209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.080702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1279.826177] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1285.950259] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1290.669502] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1296.637914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1297.030455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.204384] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1308.556584] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1309.293930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1319.367496] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 1319.914793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.923031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.932507] Lustre: Skipped 2 previous similar messages [ 1319.936089] Lustre: Skipped 1 previous similar message [ 1320.141208] Lustre: server umount lustre-MDT0000 complete [ 1323.975381] LustreError: 39955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964459 with bad export cookie 3130933335155293088 [ 1323.982196] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.992379] LustreError: 39955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1324.455067] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.478732] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 1324.482266] Lustre: Skipped 1 previous similar message [ 1325.033920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.057302] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.057684] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1329.078350] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1329.086207] Lustre: Skipped 1 previous similar message [ 1334.179978] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.181177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1334.204463] LustreError: 39971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1334.246913] Lustre: Skipped 1 previous similar message [ 1338.336146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1338.791338] Lustre: server umount lustre-MDT0001 complete [ 1353.355355] Lustre: server umount lustre-OST0000 complete [ 1369.241843] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1372.598292] Key type lgssc unregistered [ 1372.919555] LNet: 43841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.934433] LNetError: 43841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.962197] LNet: Removed LNI 192.168.204.133@tcp [ 1374.122568] Key type .llcrypt unregistered [ 1374.127126] Key type ._llcrypt unregistered [ 1399.060937] Key type ._llcrypt registered [ 1399.067841] Key type .llcrypt registered [ 1399.166853] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1399.981038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.124664] alg: No test for adler32 (adler32-zlib) [ 1401.182077] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1401.386823] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1403.064630] Key type lgssc registered [ 1403.973584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1410.055760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1410.066691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.428622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.502641] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1421.797946] LustreError: 45252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.823521] LustreError: 45252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1423.926347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.383188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1429.307685] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1438.268739] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1443.641648] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1450.570773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1450.963176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.140477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1456.647984] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1463.301442] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1475.555639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.556061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1475.576260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1476.537538] Lustre: lustre-OST0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 1476.543796] Lustre: Skipped 1 previous similar message [ 1479.752749] Lustre: server umount lustre-OST0000 complete [ 1500.130348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.143157] Lustre: Skipped 1 previous similar message [ 1500.148570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.155324] Lustre: Skipped 1 previous similar message [ 1503.863088] Lustre: server umount lustre-MDT0000 complete [ 1505.255958] LustreError: 46422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1505.295900] LustreError: 46422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1507.558886] LustreError: 45235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964642 with bad export cookie 18403084933888708903 [ 1507.563538] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.567602] LustreError: 45235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1513.956950] Lustre: server umount lustre-MDT0001 complete [ 1526.569222] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1530.187484] Key type lgssc unregistered [ 1530.591304] LNet: 48960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.620439] LNetError: 48960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.637714] LNet: Removed LNI 192.168.204.133@tcp [ 1531.568965] Key type .llcrypt unregistered [ 1531.572249] Key type ._llcrypt unregistered [ 1560.084407] Key type ._llcrypt registered [ 1560.091029] Key type .llcrypt registered [ 1560.291143] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1562.017826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.101324] alg: No test for adler32 (adler32-zlib) [ 1563.548791] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1563.984432] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1565.744286] Key type lgssc registered [ 1566.722506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1577.263497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1577.276801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.848727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.838616] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1588.195316] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.208285] LustreError: 50374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1593.314270] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.657874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.412730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1600.785589] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1611.023629] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1619.644649] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1629.190727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.179267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.221868] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1644.812624] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1648.634220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1654.270215] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1657.981546] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1670.671942] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-10-0214h13m23s [ 1672.278214] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1676.258613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1676.278764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.305944] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 1676.327092] Lustre: Skipped 3 previous similar messages [ 1677.346679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1677.369289] Lustre: Skipped 2 previous similar messages [ 1680.703857] Lustre: server umount lustre-MDT0000 complete [ 1681.453105] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1685.036980] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1687.973141] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.996915] LustreError: 50369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1694.138649] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1694.172627] LustreError: 50371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1694.286080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.519365] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1694.912800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1695.012509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1695.931744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1700.164874] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1703.909563] Lustre: DEBUG MARKER: Wait for df (36676) ... [ 1705.724614] Lustre: DEBUG MARKER: done [ 1710.586178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.594512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.621363] Lustre: Skipped 2 previous similar messages [ 1710.631852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1710.638733] Lustre: Skipped 1 previous similar message [ 1716.715927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.732016] Lustre: Skipped 3 previous similar messages [ 1721.825898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.841823] Lustre: Skipped 1 previous similar message [ 1724.384200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1724.583274] Lustre: server umount lustre-OST0000 complete [ 1732.084724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.103654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.109080] Lustre: Skipped 2 previous similar messages [ 1732.116161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.131568] LustreError: Skipped 1 previous similar message [ 1732.160963] Lustre: Skipped 1 previous similar message [ 1746.403871] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1746.717335] Lustre: server umount lustre-MDT0000 complete [ 1751.842770] LustreError: 50357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964886 with bad export cookie 814831557313947869 [ 1751.853076] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1752.085155] Lustre: server umount lustre-MDT0001 complete [ 1767.210120] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1770.411581] Key type lgssc unregistered [ 1770.865787] LNet: 55308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.876591] LNetError: 55308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.938122] LNet: Removed LNI 192.168.204.133@tcp [ 1772.059116] Key type .llcrypt unregistered [ 1772.061554] Key type ._llcrypt unregistered [ 1817.164833] Key type ._llcrypt registered [ 1817.167723] Key type .llcrypt registered [ 1817.335624] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1819.768905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1819.859901] alg: No test for adler32 (adler32-zlib) [ 1821.111533] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1821.541532] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1823.288918] Key type lgssc registered [ 1824.835680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.889126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1833.918058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.341181] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1835.383118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1835.561934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.375659] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1851.186525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.352909] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1851.398394] Lustre: 57531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1851.872813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1858.964896] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1867.999664] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1876.521944] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1884.715298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1884.935631] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1885.081984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.016083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1892.528832] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1901.691872] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1910.869603] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-10-0214h17m23s [ 1913.084517] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1925.258205] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1929.768589] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1930.018876] LustreError: 59059:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1950.216114] Lustre: lustre-MDT0000: Client 4b5c438e-f985-4b24-b588-f17895a9dad1 (at 192.168.204.33@tcp) reconnecting [ 1970.040410] LustreError: 59059:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1972.836846] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1976.032843] Lustre: DEBUG MARKER: done [ 1983.459764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.464794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.482530] LustreError: Skipped 1 previous similar message [ 1983.487819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.511088] Lustre: Skipped 1 previous similar message [ 1988.587983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.598560] Lustre: Skipped 2 previous similar messages [ 1993.709204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.714149] Lustre: Skipped 1 previous similar message [ 1997.281516] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1997.678448] Lustre: server umount lustre-OST0000 complete [ 2003.940194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.947192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.975808] Lustre: Skipped 1 previous similar message [ 2003.998741] Lustre: Skipped 1 previous similar message [ 2011.112755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.137793] Lustre: Skipped 3 previous similar messages [ 2017.250566] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2017.814978] Lustre: server umount lustre-MDT0000 complete [ 2022.772362] LustreError: 60294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965157 with bad export cookie 139143993508559103 [ 2022.774812] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.781364] LustreError: 60294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2023.116362] Lustre: server umount lustre-MDT0001 complete [ 2037.031898] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2040.978128] Key type lgssc unregistered [ 2041.432639] LNet: 61867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.441149] LNetError: 61867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.474260] LNet: Removed LNI 192.168.204.133@tcp [ 2042.445881] Key type .llcrypt unregistered [ 2042.454552] Key type ._llcrypt unregistered [ 2096.479845] Key type ._llcrypt registered [ 2096.482743] Key type .llcrypt registered [ 2096.581805] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2098.271269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2098.331222] alg: No test for adler32 (adler32-zlib) [ 2099.466576] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2099.830740] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2101.656637] Key type lgssc registered [ 2102.953331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.409283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2107.441426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.912352] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2108.945408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2109.225184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2110.836803] Lustre: Failing over lustre-MDT0000 [ 2111.465677] Lustre: server umount lustre-MDT0000 complete [ 2121.776321] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:20:54 (1790965254) [ 2129.592781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.220976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.069447] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2146.856838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.978549] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2147.024291] Lustre: 64664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2147.448239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2153.254487] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2162.456779] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2170.226766] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.327891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.609784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2179.923647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.412171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2187.990574] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2197.623675] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2204.940170] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2213.858796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2213.871947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.913269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2216.423971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.425424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.722115] Lustre: server umount lustre-MDT0000 complete [ 2222.586054] LustreError: 66523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965357 with bad export cookie 1429091938748632897 [ 2222.593297] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.598540] LustreError: 66523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2223.084827] Lustre: server umount lustre-MDT0001 complete [ 2226.717980] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2232.340548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2242.016361] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965361/real 1790965361] req@000000004f8d9359 x1877963161251072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965377 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2242.088501] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2242.104853] Lustre: Skipped 1 previous similar message [ 2242.281614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.042567] Lustre: 63587:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965362/real 1790965362] req@0000000063708c91 x1877963161251200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2247.136238] Lustre: 63586:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965366/real 1790965366] req@000000009b528f22 x1877963161251328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2247.269930] LustreError: 63583:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000050d4ee16 x1877963161252096/t0(0) o250->MGC192.168.204.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2247.867209] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.904098] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2248.056839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2248.277462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2251.111173] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.148026] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2252.135206] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.160328] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2253.155580] Lustre: 63584:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965372/real 1790965372] req@0000000080f57cb4 x1877963161251712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965388 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2253.172710] Lustre: 63584:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2254.138609] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2257.391758] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.424614] LustreError: 68440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2262.502174] LustreError: 68439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.513964] LustreError: 68439:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2264.626958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.069180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2270.197270] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2270.704315] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2271.312815] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2279.300684] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2286.382214] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2295.777683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2295.786525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.821829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.828935] Lustre: Skipped 2 previous similar messages [ 2301.921536] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.937949] LustreError: 68441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2303.381599] Lustre: server umount lustre-MDT0000 complete [ 2307.831862] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965442 with bad export cookie 1429091938748635172 [ 2307.833218] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2307.843932] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2308.207460] Lustre: server umount lustre-MDT0001 complete [ 2316.199697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2327.913434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.544142] Lustre: 63586:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965447/real 1790965447] req@0000000026a34986 x1877963161283456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2328.604109] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.621284] Lustre: Skipped 2 previous similar messages [ 2332.772648] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x13d5278bf07c9d87 [ 2332.799977] Lustre: MGC192.168.204.133@tcp: Connection restored to 0@lo (at 0@lo) [ 2333.237432] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.265601] LustreError: 66518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2333.340890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.428591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2337.456910] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2338.453260] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2338.560429] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965457/real 1790965457] req@00000000c72ebced x1877963161283840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790965473 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2338.601544] Lustre: 63585:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2349.905407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.714084] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2356.349996] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2365.056995] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2371.864656] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2386.408580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.414893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.422075] Lustre: Skipped 2 previous similar messages [ 2386.423862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.435192] Lustre: Skipped 5 previous similar messages [ 2388.274541] Lustre: server umount lustre-MDT0000 complete [ 2391.523043] LustreError: 71126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.543243] LustreError: 71126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2393.874900] LustreError: 64656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965528 with bad export cookie 1429091938748636551 [ 2393.910379] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.912911] LustreError: 64656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2394.903404] Lustre: server umount lustre-MDT0001 complete [ 2405.230809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2409.277857] Lustre: DEBUG MARKER: files compared the same [ 2422.008619] Lustre: server umount lustre-OST0000 complete [ 2437.000067] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2440.191299] Key type lgssc unregistered [ 2440.492956] LNet: 74404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.501282] LNetError: 74404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.519711] LNet: Removed LNI 192.168.204.133@tcp [ 2441.294127] Key type .llcrypt unregistered [ 2441.300143] Key type ._llcrypt unregistered [ 2469.252680] Key type ._llcrypt registered [ 2469.256762] Key type .llcrypt registered [ 2469.423727] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2471.225427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2471.332027] alg: No test for adler32 (adler32-zlib) [ 2472.855143] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2473.138743] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2474.872258] Key type lgssc registered [ 2476.446096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.133453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.196412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.875363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.932521] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all [ 2496.993297] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.037171] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2502.113851] LustreError: 75828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.080403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.571306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2512.122281] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all [ 2521.538086] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.882744] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2536.809062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2537.359358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.692617] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all [ 2552.818179] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2557.811578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2567.136870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.156921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.181796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.167315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.173290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2568.186659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.452235] Lustre: server umount lustre-OST0000 complete [ 2578.415275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2578.428726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.437218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.437229] Lustre: Skipped 1 previous similar message [ 2578.481698] Lustre: Skipped 1 previous similar message [ 2583.524041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.546389] Lustre: Skipped 2 previous similar messages [ 2584.141034] Lustre: server umount lustre-MDT0000 complete [ 2588.646507] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.682097] LustreError: 75827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2588.920448] LustreError: 75808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965723 with bad export cookie 11605605082854668335 [ 2588.921206] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.931517] LustreError: 75808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2589.369633] Lustre: server umount lustre-MDT0001 complete [ 2602.259506] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2605.801974] Key type lgssc unregistered [ 2606.224560] LNet: 79360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.231627] LNetError: 79360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.259965] LNet: Removed LNI 192.168.204.133@tcp [ 2607.050617] Key type .llcrypt unregistered [ 2607.054494] Key type ._llcrypt unregistered [ 2628.603673] Key type ._llcrypt registered [ 2628.607519] Key type .llcrypt registered [ 2628.744893] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2629.795477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2630.001419] alg: No test for adler32 (adler32-zlib) [ 2631.224880] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2631.709491] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2633.416233] Key type lgssc registered [ 2634.650862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2643.323560] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2643.347466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.914636] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2664.033379] LustreError: 80727:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 4: rc = -110 [ 2690.848154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2697.881607] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2706.202048] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2716.304224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.398783] Lustre: *** cfs_fail_loc=706, val=0*** [ 2716.913065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2717.034811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2722.128522] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2722.281365] LustreError: 80752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.298982] LustreError: 80752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2725.365393] LustreError: 81815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.406667] LustreError: 81815:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2730.467895] LustreError: 81814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.500533] LustreError: 81814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2730.972499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.469148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2735.633762] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2743.466317] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2749.064524] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2751.970625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2751.976709] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.990380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.000953] Lustre: Skipped 1 previous similar message [ 2752.012976] Lustre: Skipped 1 previous similar message [ 2757.093109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2757.103814] Lustre: Skipped 1 previous similar message [ 2757.477160] Lustre: server umount lustre-OST0000 complete [ 2762.211037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.212502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.233462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.239679] Lustre: Skipped 1 previous similar message [ 2762.257132] Lustre: Skipped 1 previous similar message [ 2767.330602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.335278] Lustre: Skipped 1 previous similar message [ 2767.559804] Lustre: server umount lustre-MDT0000 complete [ 2772.078774] LustreError: 80734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965907 with bad export cookie 12374581135465982392 [ 2772.100257] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.100502] LustreError: 80734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2772.458412] Lustre: server umount lustre-MDT0001 complete [ 2783.593791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2786.362267] Key type lgssc unregistered [ 2786.673292] LNet: 84344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.684591] LNetError: 84344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.712153] LNet: Removed LNI 192.168.204.133@tcp [ 2787.607655] Key type .llcrypt unregistered [ 2787.609609] Key type ._llcrypt unregistered [ 2811.577985] Key type ._llcrypt registered [ 2811.580301] Key type .llcrypt registered [ 2811.704508] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2812.696129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2812.838288] alg: No test for adler32 (adler32-zlib) [ 2813.905880] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 2814.220413] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2815.888334] Key type lgssc registered [ 2817.350306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2826.462476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2826.484257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.820493] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2841.111797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.635607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.196251] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2846.692907] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2846.711484] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2851.812081] LustreError: 86452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2851.839401] LustreError: 86452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2855.100458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.418571] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.447847] LustreError: 86448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2855.466788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.515292] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.542683] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2861.615065] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2865.639107] LustreError: 87193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.655416] LustreError: 87193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2870.810879] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.820474] Lustre: MGS: Client 6c0baa21-1de3-43e8-8773-7676bec66615 (at 0@lo) reconnecting [ 2870.857164] LustreError: 86462:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2871.130612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2871.233943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2876.086585] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2891.746915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.766142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.771657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2891.787440] Lustre: Skipped 1 previous similar message [ 2891.804640] Lustre: Skipped 1 previous similar message [ 2896.870809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2897.317499] Lustre: server umount lustre-OST0000 complete [ 2901.998336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.009645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2902.031231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2902.072882] Lustre: Skipped 2 previous similar messages [ 2907.108685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2907.129648] Lustre: Skipped 1 previous similar message [ 2907.655988] Lustre: server umount lustre-MDT0000 complete [ 2911.249934] Lustre: server umount lustre-MDT0001 complete [ 2915.013897] Lustre: server umount MGS complete [ 2926.736875] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:34:20 (1790966060) [ 2958.844342] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 2968.902722] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3006.346876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.998400] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3024.013623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.522098] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3024.577457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3024.675083] Lustre: lustre-MDT0001: new disk, initializing [ 3024.781539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.821308] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3024.837365] Lustre: 93631:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3024.855623] LustreError: 93631:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3029.449643] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3041.280453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3041.634279] Lustre: lustre-OST0000: new disk, initializing [ 3041.638328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3041.652061] Lustre: 94398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3041.749248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.196418] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3050.039389] Lustre: 94735:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ca33a2a9 x1877963910979456/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790966201 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3060.766646] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3068.077826] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.089927] Lustre: MGS: Client af9fca87-322a-40ce-ad89-2621999e0001 (at 0@lo) reconnecting [ 3068.126770] Lustre: 92899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3068.140909] LustreError: 93617:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3068.167558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3068.262525] Lustre: lustre-MDT0000: new disk, initializing [ 3068.403626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.436343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3068.473094] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3069.538994] Lustre: 95469:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000007857193 x1877963910986368/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1790966220 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3073.305708] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3074.656598] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3077.606813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3077.613716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3078.633738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3078.639451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3078.702690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3103.203795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3103.211393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.220345] Lustre: Skipped 1 previous similar message [ 3103.222993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.921471] Lustre: server umount lustre-OST0000 complete [ 3112.422121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3112.427839] LustreError: Skipped 1 previous similar message [ 3112.448159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.469678] Lustre: Skipped 1 previous similar message [ 3112.494618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.504259] Lustre: Skipped 3 previous similar messages [ 3117.037184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.689918] Lustre: server umount lustre-MDT0000 complete [ 3128.510822] Lustre: server umount lustre-MDT0001 complete [ 3132.228577] Lustre: server umount MGS complete [ 3146.610933] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:38:00 (1790966280) [ 3166.907403] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3169.624202] Key type lgssc unregistered [ 3169.895588] LNet: 97968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.902710] LNetError: 97968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3169.924051] LNet: Removed LNI 192.168.204.133@tcp [ 3170.769458] Key type .llcrypt unregistered [ 3170.771702] Key type ._llcrypt unregistered [ 3186.988543] Key type ._llcrypt registered [ 3186.990911] Key type .llcrypt registered [ 3187.117347] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3188.127866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3188.301306] alg: No test for adler32 (adler32-zlib) [ 3189.329598] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3189.559235] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3191.320125] Key type lgssc registered [ 3192.677895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3201.466398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3201.478253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.977205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.699388] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3213.295205] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.322149] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3218.408946] LustreError: 99199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.649534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.989504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3223.281127] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3230.520725] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3236.641221] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3242.792253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.088303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.244116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3249.139251] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3256.977549] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3265.001678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3265.015914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.035267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.096860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.098542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.106369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.135035] Lustre: Skipped 1 previous similar message [ 3270.652772] Lustre: server umount lustre-OST0000 complete [ 3275.233447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3275.244058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.277211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.335117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.361950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.384108] Lustre: Skipped 1 previous similar message [ 3283.226696] Lustre: server umount lustre-MDT0000 complete [ 3284.449996] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.481719] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3287.808741] LustreError: 99933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966422 with bad export cookie 15088783424123447828 [ 3287.815389] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.829544] LustreError: 99933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3288.165804] Lustre: server umount lustre-MDT0001 complete [ 3302.156602] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3306.096099] Key type lgssc unregistered [ 3306.553440] LNet: 102730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.575272] LNetError: 102730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.608877] LNet: Removed LNI 192.168.204.133@tcp [ 3307.675541] Key type .llcrypt unregistered [ 3307.678557] Key type ._llcrypt unregistered [ 3328.461876] Key type ._llcrypt registered [ 3328.465583] Key type .llcrypt registered [ 3328.644434] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3329.796558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.849777] alg: No test for adler32 (adler32-zlib) [ 3331.042768] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3331.261667] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3333.040206] Key type lgssc registered [ 3334.576664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3342.614945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3342.624639] LustreError: 103811:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3342.634733] LustreError: 103812:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3342.648820] LustreError: 103811:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3342.661703] LustreError: 103811:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3342.672504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.673835] LustreError: 103811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3344.448576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.009380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.284053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3359.857517] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3368.107802] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing lsmod [ 3372.258934] LustreError: 105013:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3372.311933] LustreError: 105020:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3372.317970] LustreError: 105013:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 3372.328351] LustreError: 105013:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3372.337264] LustreError: 105013:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3372.344671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.904423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.150122] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.164152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3378.168810] Lustre: Skipped 1 previous similar message [ 3379.239954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3379.249858] Lustre: Skipped 4 previous similar messages [ 3383.087295] Lustre: server umount lustre-MDT0000 complete [ 3384.293982] LustreError: 103849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3384.309661] LustreError: 103849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3387.990092] LustreError: 103826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966523 with bad export cookie 16555329348129514757 [ 3387.994917] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3388.003230] LustreError: 103826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3389.410385] LustreError: 103844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.411192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.447667] Lustre: Skipped 1 previous similar message [ 3389.453196] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3389.600384] LustreError: 104282:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001962c913 x1877964453075456/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3389.840617] Lustre: server umount lustre-MDT0001 complete [ 3394.598478] Lustre: server umount lustre-OST0000 complete [ 3403.985287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3404.434308] LustreError: 106084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.527764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.368694] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3409.894290] LustreError: 106085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.991267] LustreError: 106084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.820425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3418.528413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3423.155648] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3430.955481] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3436.479476] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3443.694832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.105364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.202364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3450.385726] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3458.529973] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3472.877355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3472.890454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.907113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3473.897888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.011449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3479.021194] Lustre: Skipped 2 previous similar messages [ 3484.138790] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3484.334448] Lustre: server umount lustre-OST0000 complete [ 3489.250670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.279810] Lustre: Skipped 1 previous similar message [ 3489.287664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.299705] Lustre: Skipped 4 previous similar messages [ 3495.095966] Lustre: server umount lustre-MDT0000 complete [ 3499.502779] LustreError: 106084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.540125] LustreError: 106084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3499.612644] LustreError: 106065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966634 with bad export cookie 16555329348129515457 [ 3499.615502] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3499.629225] LustreError: 106065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3500.345454] Lustre: server umount lustre-MDT0001 complete [ 3514.263575] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3517.271296] Key type lgssc unregistered [ 3517.494185] LNet: 109624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3517.502417] LNetError: 109624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3517.514570] LNet: Removed LNI 192.168.204.133@tcp [ 3518.520266] Key type .llcrypt unregistered [ 3518.523664] Key type ._llcrypt unregistered [ 3547.005370] Key type ._llcrypt registered [ 3547.008257] Key type .llcrypt registered [ 3547.132510] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3549.167467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3549.339270] alg: No test for adler32 (adler32-zlib) [ 3550.553872] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3551.075123] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3552.929690] Key type lgssc registered [ 3554.767182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3567.028820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3567.050192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.505220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3574.340631] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3578.855317] LustreError: 111042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.880956] LustreError: 111042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3582.136861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.404542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3586.624851] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3594.642752] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3601.707665] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3611.119075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.482627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.558967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3619.016375] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3628.794844] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3633.868804] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3636.586451] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3643.366749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3643.373066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.381535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3643.410763] Lustre: Skipped 1 previous similar message [ 3647.058719] Lustre: server umount lustre-OST0000 complete [ 3653.618930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3653.628264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3653.635885] Lustre: Skipped 1 previous similar message [ 3653.653785] Lustre: Skipped 2 previous similar messages [ 3658.721929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3658.744107] Lustre: Skipped 1 previous similar message [ 3659.135120] Lustre: server umount lustre-MDT0000 complete [ 3663.856688] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3663.925360] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3664.700225] LustreError: 111024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966799 with bad export cookie 3578860667976797568 [ 3664.703975] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3664.743828] LustreError: 111024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3665.626480] Lustre: server umount lustre-MDT0001 complete [ 3683.593920] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3687.042966] Key type lgssc unregistered [ 3687.535551] LNet: 114693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3687.562794] LNetError: 114693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3687.593947] LNet: Removed LNI 192.168.204.133@tcp [ 3688.857777] Key type .llcrypt unregistered [ 3688.861839] Key type ._llcrypt unregistered [ 3712.562064] Key type ._llcrypt registered [ 3712.563799] Key type .llcrypt registered [ 3712.654651] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3713.751876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3713.884474] alg: No test for adler32 (adler32-zlib) [ 3715.164844] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3715.666457] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3717.424408] Key type lgssc registered [ 3719.436409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3730.333764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3730.380312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.413134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.072231] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3742.690514] LustreError: 115926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.740068] LustreError: 115926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3747.424815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3747.813264] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3748.099134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3752.658651] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3761.006890] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3766.968625] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3774.229834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3774.681108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.838479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3781.704843] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3792.748475] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3803.626220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3803.626296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.630381] LustreError: Skipped 1 previous similar message [ 3803.630728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3803.664593] Lustre: Skipped 1 previous similar message [ 3808.769437] Lustre: server umount lustre-OST0000 complete [ 3813.858100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.882733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.902100] Lustre: Skipped 2 previous similar messages [ 3818.977838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3818.995192] Lustre: Skipped 1 previous similar message [ 3819.274107] Lustre: server umount lustre-MDT0000 complete [ 3823.074133] LustreError: 115926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.074133] LustreError: 115923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3824.050667] LustreError: 115908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966959 with bad export cookie 16204645552497725620 [ 3824.059159] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3824.076911] LustreError: 115908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3824.537802] Lustre: server umount lustre-MDT0001 complete [ 3836.914367] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3841.393932] Key type lgssc unregistered [ 3841.770630] LNet: 119460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3841.794308] LNetError: 119460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3841.824467] LNet: Removed LNI 192.168.204.133@tcp [ 3843.027170] Key type .llcrypt unregistered [ 3843.029202] Key type ._llcrypt unregistered [ 3871.476478] Key type ._llcrypt registered [ 3871.478858] Key type .llcrypt registered [ 3871.657822] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3872.951654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3872.976302] alg: No test for adler32 (adler32-zlib) [ 3874.366222] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3874.788674] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3876.569110] Key type lgssc registered [ 3878.049176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3887.626372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3887.657044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.584340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.289476] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3899.877535] LustreError: 120871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.901399] LustreError: 120871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3904.997587] LustreError: 120872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.313133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3906.807172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3911.925897] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3921.202922] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3928.149918] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.704884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.257913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.834286] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3955.320890] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3959.763877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3968.550615] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3973.631736] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3973.642990] Lustre: Skipped 2 previous similar messages [ 3975.794597] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3975.844446] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3979.976972] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3979.985824] Lustre: Skipped 2 previous similar messages [ 3983.995745] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3984.003192] Lustre: Skipped 1 previous similar message [ 3992.714443] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3992.733344] Lustre: Skipped 2 previous similar messages [ 3998.329812] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp [ 4003.988849] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.33/24@tcp in log lustre-MDT0000 [ 4003.996968] Lustre: Skipped 2 previous similar messages [ 4005.654511] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.33/24@tcp [ 4005.665863] Lustre: Skipped 1 previous similar message [ 4013.025882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.040443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.042214] LustreError: Skipped 1 previous similar message [ 4013.044369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.072940] Lustre: Skipped 1 previous similar message [ 4016.101309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4016.112705] Lustre: Skipped 2 previous similar messages [ 4021.219600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.229260] Lustre: Skipped 1 previous similar message [ 4026.336187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4026.361671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.389436] Lustre: Skipped 1 previous similar message [ 4026.576955] Lustre: server umount lustre-OST0000 complete [ 4031.459135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.468770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.484720] Lustre: Skipped 1 previous similar message [ 4031.495912] Lustre: Skipped 1 previous similar message [ 4037.541809] Lustre: server umount lustre-MDT0000 complete [ 4041.466747] LustreError: 120852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967176 with bad export cookie 14150815488815130417 [ 4041.480306] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4041.491981] LustreError: 120852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4042.078699] Lustre: server umount lustre-MDT0001 complete [ 4052.246705] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4054.852487] Key type lgssc unregistered [ 4055.349492] LNet: 125724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.360038] LNetError: 125724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4055.373495] LNet: Removed LNI 192.168.204.133@tcp [ 4056.429954] Key type .llcrypt unregistered [ 4056.434501] Key type ._llcrypt unregistered [ 4094.493985] Key type ._llcrypt registered [ 4094.495984] Key type .llcrypt registered [ 4094.626838] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4096.239450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4096.299179] alg: No test for adler32 (adler32-zlib) [ 4098.397498] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4098.850617] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4100.652741] Key type lgssc registered [ 4102.860530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4113.139463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4113.151576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.639376] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4114.679484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4114.685182] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.33/24@tcp [ 4119.654786] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4124.132428] LustreError: 127281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4124.155193] LustreError: 127281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4129.249958] LustreError: 127280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4130.805146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4131.158269] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4131.177362] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.33/24@tcp [ 4131.223288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.133989] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4147.632254] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4155.836763] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4163.672111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4164.163660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.975640] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4179.330945] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4184.649290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4194.788100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.791327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.799202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.813048] Lustre: Skipped 1 previous similar message [ 4194.832858] Lustre: Skipped 1 previous similar message [ 4199.110817] Lustre: server umount lustre-OST0000 complete [ 4205.026053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.053397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.061080] Lustre: Skipped 1 previous similar message [ 4210.033951] Lustre: server umount lustre-MDT0000 complete [ 4210.165800] LustreError: 127281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.178389] LustreError: 127281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4214.248426] LustreError: 127280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.290273] LustreError: 127280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4214.676766] LustreError: 127262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967349 with bad export cookie 1371367246886466096 [ 4214.697147] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4214.701809] LustreError: 127262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.150892] Lustre: server umount lustre-MDT0001 complete [ 4228.253915] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4232.079573] Key type lgssc unregistered [ 4232.509864] LNet: 130869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4232.521107] LNetError: 130869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4232.547377] LNet: Removed LNI 192.168.204.133@tcp [ 4233.755889] Key type .llcrypt unregistered [ 4233.757945] Key type ._llcrypt unregistered [ 4272.266262] Key type ._llcrypt registered [ 4272.268953] Key type .llcrypt registered [ 4272.497495] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4294.334362] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4295.580166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4295.764926] alg: No test for adler32 (adler32-zlib) [ 4297.043144] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4297.268856] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4298.897055] Key type lgssc registered [ 4300.360441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4336.627428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4336.685219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.924514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4337.947652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4338.030582] Lustre: lustre-MDT0000: new disk, initializing [ 4338.131710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.164814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4342.156775] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4353.363488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.545481] Lustre: 134290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4353.594370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4353.601123] Lustre: Skipped 1 previous similar message [ 4353.650311] Lustre: lustre-MDT0001: new disk, initializing [ 4353.739442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4353.797956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4353.813497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4358.322371] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4368.664253] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4376.166909] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4383.933049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4384.464110] Lustre: lustre-OST0000: new disk, initializing [ 4384.473386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4384.496576] Lustre: 136256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4384.680268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.684983] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4394.056122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4394.073350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4394.228242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4403.006122] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4408.204158] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4408.556987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4413.755262] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4413.976620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4420.077173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.091516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4420.117885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4420.120019] LustreError: Skipped 1 previous similar message [ 4420.160977] Lustre: Skipped 1 previous similar message [ 4422.387401] Lustre: server umount lustre-OST0000 complete [ 4430.308797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.329170] Lustre: Skipped 1 previous similar message [ 4430.346442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4430.351021] Lustre: Skipped 1 previous similar message [ 4435.428055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.439522] Lustre: Skipped 1 previous similar message [ 4440.546204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.560130] Lustre: Skipped 1 previous similar message [ 4441.058811] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4441.506845] Lustre: server umount lustre-MDT0000 complete [ 4447.354275] LustreError: 134282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967582 with bad export cookie 1120879926010555166 [ 4447.361714] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.378201] LustreError: 134282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4447.784941] Lustre: server umount lustre-MDT0001 complete [ 4457.162570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.679896] LustreError: 138474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.725227] LustreError: 138474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4457.839552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.077085] LustreError: 138475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.563582] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4467.184419] LustreError: 138474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.298904] LustreError: 138475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.394792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.083438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4481.451123] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4491.712444] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.847726] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4506.116521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4506.790818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4513.788619] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4521.917228] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4527.728485] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4528.085872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4533.667902] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4533.907921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4544.005111] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4544.246429] Lustre: lustre-OST0001: new disk, initializing [ 4544.250845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4544.255813] Lustre: 141858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4544.343054] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4551.066922] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4553.360735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4553.379536] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4553.485679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4561.243727] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4561.707779] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4568.030552] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4568.613395] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4578.997268] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8964d1aa6800.ost_server_uuid 50 [ 4581.726793] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8964d1aa6800.ost_server_uuid in IDLE state after 0 sec [ 4594.644305] Lustre: server umount lustre-OST0001 complete [ 4594.662384] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4594.671575] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.681236] LustreError: Skipped 1 previous similar message [ 4594.701469] Lustre: Skipped 1 previous similar message [ 4594.706381] LustreError: 140298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.778419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.826209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4604.385950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4604.396059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.905424] LustreError: 140297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.952487] LustreError: 140297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4605.897704] Lustre: server umount lustre-OST0000 complete [ 4610.022051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4610.025235] Lustre: Skipped 3 previous similar messages [ 4624.356139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4624.912539] Lustre: server umount lustre-MDT0000 complete [ 4625.384736] LustreError: 138471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.413382] LustreError: 138471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4629.202287] LustreError: 138456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967764 with bad export cookie 1120879926010556055 [ 4629.213711] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4629.222885] LustreError: 138456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4629.806228] Lustre: server umount lustre-MDT0001 complete [ 4653.548430] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4657.754991] Key type lgssc unregistered [ 4658.014287] LNet: 144869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4658.028552] LNetError: 144869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4658.062953] LNet: Removed LNI 192.168.204.133@tcp [ 4659.191064] Key type .llcrypt unregistered [ 4659.194163] Key type ._llcrypt unregistered [ 4710.110567] Key type ._llcrypt registered [ 4710.112896] Key type .llcrypt registered [ 4710.217175] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4729.243075] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4730.362679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4730.433522] alg: No test for adler32 (adler32-zlib) [ 4731.699486] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4732.035368] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4733.744317] Key type lgssc registered [ 4735.210403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4786.803741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4786.882515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.093167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4788.117306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4788.165805] Lustre: lustre-MDT0000: new disk, initializing [ 4788.223152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.235456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4793.053878] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4805.329303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.501921] Lustre: 149020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4805.569441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4805.577380] Lustre: Skipped 1 previous similar message [ 4805.696472] Lustre: lustre-MDT0001: new disk, initializing [ 4805.880085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4805.975676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4805.997395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4811.662236] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4821.932860] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4827.639327] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4834.619161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4834.959632] Lustre: lustre-OST0000: new disk, initializing [ 4834.962689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4834.968226] Lustre: 150991:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4835.056237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.217275] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4842.587577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4842.614882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4842.725820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.630332] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4857.486484] Lustre: Failing over lustre-OST0000 [ 4857.832779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4857.870061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4858.026648] Lustre: server umount lustre-OST0000 complete [ 4874.723179] Lustre: 147243:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790967993/real 1790967993] req@00000000d74d73ee x1877965922141696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790968009 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4874.769851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.842188] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790967998/real 1790967998] req@00000000c4e52029 x1877965922143488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790968014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4880.871765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4881.131992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4881.154974] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4884.198528] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4885.223745] Lustre: 147244:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790968003/real 1790968003] req@00000000aa776be6 x1877965922144512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790968019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4887.709806] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4889.379090] Lustre: 147244:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790968008/real 1790968008] req@00000000da55c6c5 x1877965922145536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790968024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4892.474857] Lustre: Failing over lustre-MDT0000 [ 4892.730886] Lustre: server umount lustre-MDT0000 complete [ 4893.159090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4893.165204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.187453] LustreError: 149032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4893.202961] LustreError: 149032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4894.696751] LustreError: 149027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.718290] LustreError: 149027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4899.809345] LustreError: 150199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.841445] LustreError: 150199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4904.930047] LustreError: 149032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.955335] LustreError: 149032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4910.048130] Lustre: 147243:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790968029/real 1790968029] req@00000000ed15f37d x1877965922155648/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790968045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4910.055982] LustreError: 150199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.097772] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4910.142203] LustreError: 150199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4913.719953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4920.291045] LustreError: 147241:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000796c3924 x1877965922159104/t0(0) o250->MGC192.168.204.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4920.326411] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4920.349814] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4920.524055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4920.600641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.682831] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4921.258115] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4925.513175] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4925.933198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4925.990942] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4926.010584] Lustre: Skipped 1 previous similar message [ 4926.090057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4936.171300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4936.178429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.180087] LustreError: Skipped 1 previous similar message [ 4936.210022] Lustre: Skipped 2 previous similar messages [ 4936.216288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4936.248660] Lustre: Skipped 1 previous similar message [ 4940.614872] Lustre: server umount lustre-OST0000 complete [ 4946.403839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4946.411450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.432699] Lustre: Skipped 1 previous similar message [ 4946.439101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4946.445800] Lustre: Skipped 1 previous similar message [ 4951.023519] Lustre: server umount lustre-MDT0000 complete [ 4951.521451] LustreError: 149033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.534038] LustreError: 149033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4954.898891] LustreError: 149012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968089 with bad export cookie 16826439346800249605 [ 4954.903459] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4954.916134] LustreError: 149012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4955.314869] Lustre: server umount lustre-MDT0001 complete [ 4966.210387] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4969.080289] Key type lgssc unregistered [ 4969.330697] LNet: 154647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4969.337941] LNetError: 154647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4970.406314] LNet: Removed LNI 192.168.204.133@tcp [ 4971.383441] Key type .llcrypt unregistered [ 4971.388342] Key type ._llcrypt unregistered [ 5013.216841] Key type ._llcrypt registered [ 5013.222332] Key type .llcrypt registered [ 5013.413590] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5014.981246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5015.234573] alg: No test for adler32 (adler32-zlib) [ 5016.486957] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 5016.789929] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5018.611981] Key type lgssc registered [ 5020.238876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5031.660829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5031.678824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.132452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5038.390939] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5043.693089] LustreError: 156173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.728047] LustreError: 156173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5048.801954] LustreError: 156172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.511850] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5053.407622] LustreError: 156201:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5053.425401] LustreError: 156201:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5053.436346] LustreError: 156201:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 5053.858571] Lustre: server umount lustre-MDT0000 complete [ 5068.905489] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5080.224782] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5112.687274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5112.943895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5112.968941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5113.035545] Lustre: lustre-MDT0000: new disk, initializing [ 5113.120529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5113.136472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5117.037479] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5126.701561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.881690] Lustre: 159506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5126.961587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5126.969659] Lustre: Skipped 1 previous similar message [ 5127.062581] Lustre: lustre-MDT0001: new disk, initializing [ 5127.214240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5127.251092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5127.267642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.659532] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5140.206381] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5145.850662] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5152.895191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.164242] Lustre: lustre-OST0000: new disk, initializing [ 5153.167868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5153.174574] Lustre: 161474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5153.273388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.509858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5159.534925] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5159.629308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5159.726466] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5169.281880] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5173.148689] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5175.516764] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:11:48 (1790968308) [ 5418.977854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5418.986353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.996058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.025316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.029113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.031025] Lustre: Skipped 1 previous similar message [ 5421.054933] Lustre: Skipped 1 previous similar message [ 5424.086528] Lustre: server umount lustre-MDT0000 complete [ 5426.145551] LustreError: 159517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5426.159726] LustreError: 159517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5427.538348] LustreError: 160301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968562 with bad export cookie 12823246257027202045 [ 5427.547674] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5427.548705] LustreError: 160301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5431.268900] LustreError: 159513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5431.269610] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.306927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5431.312671] Lustre: Skipped 1 previous similar message [ 5433.862068] Lustre: server umount lustre-MDT0001 complete [ 5443.457363] Lustre: server umount lustre-OST0000 complete [ 5451.648470] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5459.490808] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5489.931510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5490.121566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5490.156830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5490.241862] Lustre: lustre-MDT0000: new disk, initializing [ 5490.310712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5490.324864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5494.095178] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5503.427989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.544339] Lustre: 166134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5503.610144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5503.619709] Lustre: Skipped 1 previous similar message [ 5503.738715] Lustre: lustre-MDT0001: new disk, initializing [ 5503.844993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5503.879218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5503.885142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5508.170572] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5517.421169] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5522.207584] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5528.287695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5528.544552] Lustre: lustre-OST0000: new disk, initializing [ 5528.547892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5528.553263] Lustre: 168098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5528.637464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.434833] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5534.759788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5534.769798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5534.846932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5542.936793] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5546.911784] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5547.102922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5551.354764] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5551.549498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5555.172611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5555.173292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.191093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5555.196837] Lustre: Skipped 1 previous similar message [ 5555.217507] Lustre: Skipped 2 previous similar messages [ 5559.474748] Lustre: server umount lustre-OST0000 complete [ 5565.411684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.412576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5565.432431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5565.442565] Lustre: Skipped 1 previous similar message [ 5569.906232] Lustre: server umount lustre-MDT0000 complete [ 5570.535189] LustreError: 166145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.557617] LustreError: 166145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5573.565669] LustreError: 166126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968708 with bad export cookie 12823246257027360882 [ 5573.566304] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5573.572263] LustreError: 166126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5573.978765] Lustre: server umount lustre-MDT0001 complete [ 5582.901314] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:18:36 (1790968716) [ 5598.087914] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5607.162385] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5639.273828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.455119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5639.501144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5639.628326] Lustre: lustre-MDT0000: new disk, initializing [ 5639.766357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.785673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5643.997439] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5654.836642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5654.974615] Lustre: 172858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5655.265351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5659.306387] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5667.207424] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5671.831951] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5677.964165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.199774] Lustre: lustre-OST0000: new disk, initializing [ 5678.202796] Lustre: Skipped 1 previous similar message [ 5678.206539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5678.210392] Lustre: Skipped 2 previous similar messages [ 5678.214955] Lustre: 174835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5683.400407] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5683.770933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5683.785425] Lustre: Skipped 1 previous similar message [ 5683.789740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5683.927533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5691.326660] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5699.556531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.563412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.584399] LustreError: 174815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5699.588769] Lustre: Skipped 2 previous similar messages [ 5699.674768] Lustre: server umount lustre-OST0000 complete [ 5704.675705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.464636] Lustre: server umount lustre-MDT0000 complete [ 5713.106918] LustreError: 173658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968848 with bad export cookie 12823246257027361757 [ 5713.111991] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.123683] LustreError: 173658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5713.414447] Lustre: server umount lustre-MDT0001 complete [ 5723.533507] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:20:56 (1790968856) [ 5741.593394] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5752.151614] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5786.264782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.475579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5786.503667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5786.582557] Lustre: lustre-MDT0000: new disk, initializing [ 5786.673890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5786.680774] Lustre: Skipped 2 previous similar messages [ 5786.693126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5791.194583] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5801.288700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.431844] Lustre: 179156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5801.596076] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5801.601204] Lustre: Skipped 1 previous similar message [ 5801.786752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5806.213765] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5816.159334] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.709748] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5827.751163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.013290] Lustre: 181134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5833.662152] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5836.319185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5836.436116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5842.659931] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5851.617494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5851.621746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.627406] Lustre: Skipped 2 previous similar messages [ 5851.630695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.796574] Lustre: server umount lustre-OST0000 complete [ 5864.977607] Lustre: server umount lustre-MDT0000 complete [ 5867.496822] LustreError: 179167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.510681] LustreError: 179167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5868.237685] LustreError: 179956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969003 with bad export cookie 12823246257027362849 [ 5868.240410] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.249643] LustreError: 179956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5868.716129] Lustre: server umount lustre-MDT0001 complete [ 5879.200879] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5882.104456] Key type lgssc unregistered [ 5882.433390] LNet: 182990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5882.443380] LNetError: 182990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5882.464500] LNet: Removed LNI 192.168.204.133@tcp [ 5883.423606] Key type .llcrypt unregistered [ 5883.427871] Key type ._llcrypt unregistered [ 5906.185775] Key type ._llcrypt registered [ 5906.189984] Key type .llcrypt registered [ 5906.291814] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5907.577400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.601520] alg: No test for adler32 (adler32-zlib) [ 5908.722476] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 5908.952477] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5910.656181] Key type lgssc registered [ 5911.759079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5918.593471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5918.609248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.937892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5923.429613] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5929.449253] LustreError: 184404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.474981] LustreError: 184404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5931.450565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.791754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5935.981685] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5942.690737] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5947.654831] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5953.327107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.634901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.599780] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5964.912512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5966.272809] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5982.692144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5982.707975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.720971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5985.762703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.767051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.742193] Lustre: server umount lustre-OST0000 complete [ 5993.446211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5993.454248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.469529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.474834] Lustre: Skipped 1 previous similar message [ 5996.002267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.004906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.027578] Lustre: Skipped 1 previous similar message [ 5998.912308] Lustre: server umount lustre-MDT0000 complete [ 6001.121591] LustreError: 185130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.134207] LustreError: 185130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6002.923892] LustreError: 184383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969137 with bad export cookie 13296130412796955793 [ 6002.938448] LustreError: 184383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6002.941737] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6003.352309] Lustre: server umount lustre-MDT0001 complete [ 6014.536947] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6017.233710] Key type lgssc unregistered [ 6017.557616] LNet: 187956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6017.564160] LNetError: 187956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6017.583544] LNet: Removed LNI 192.168.204.133@tcp [ 6018.546243] Key type .llcrypt unregistered [ 6018.552083] Key type ._llcrypt unregistered [ 6041.304371] Key type ._llcrypt registered [ 6041.307207] Key type .llcrypt registered [ 6041.421092] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6042.346086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6042.360243] alg: No test for adler32 (adler32-zlib) [ 6043.409818] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6043.603492] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6045.256975] Key type lgssc registered [ 6046.382265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6054.690078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6054.718504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.140886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6060.287496] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6066.677465] LustreError: 189370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.717505] LustreError: 189370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6069.741554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6070.262478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.745808] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6082.569444] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6087.566963] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6093.693217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.047673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.335329] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6101.254869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6107.860521] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6114.761525] Lustre: lustre-OST0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 6115.809637] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6115.819932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6115.836789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6116.835064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.839340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.857895] Lustre: lustre-OST0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 6119.868347] Lustre: Skipped 1 previous similar message [ 6120.182641] Lustre: server umount lustre-OST0000 complete [ 6128.176439] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8964d351c000.ost_server_uuid 50 [ 6129.802410] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6147.042137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6147.056194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.076036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.059062] Lustre: server umount lustre-MDT0000 complete [ 6152.688511] LustreError: 189371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.722347] LustreError: 189371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6156.028278] LustreError: 189351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969291 with bad export cookie 1528411688970989245 [ 6156.058173] LustreError: 189351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6156.065595] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.555515] Lustre: server umount lustre-MDT0001 complete [ 6166.830408] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:28:20 (1790969300) [ 6174.126995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6174.830673] LustreError: 193284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6174.954360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.246733] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6180.327117] LustreError: 193285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.422512] LustreError: 193284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.788746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.208386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6192.794290] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6200.470777] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6205.525121] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6212.279200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.548428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.533017] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6219.690116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6225.775642] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6233.127594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6233.454544] Lustre: lustre-OST0001: new disk, initializing [ 6233.457919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6233.463059] Lustre: 196123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6233.555721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6239.729278] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6243.408865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6243.429154] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6243.587633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6248.803203] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6254.058899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.062377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.081632] Lustre: Skipped 2 previous similar messages [ 6254.097214] Lustre: Skipped 2 previous similar messages [ 6257.503969] Lustre: server umount lustre-OST0000 complete [ 6259.183160] LustreError: 195103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6259.204289] LustreError: 195103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6263.434328] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6263.660497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6267.913833] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6268.187661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6279.657470] LustreError: 195539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.683574] LustreError: 195539:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6284.776900] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.780867] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6284.806151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6284.806161] Lustre: Skipped 1 previous similar message [ 6284.843198] LustreError: Skipped 1 previous similar message [ 6286.952723] Lustre: server umount lustre-OST0001 complete [ 6295.009823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.027407] Lustre: Skipped 2 previous similar messages [ 6305.248140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6305.447326] Lustre: server umount lustre-MDT0000 complete [ 6308.646256] LustreError: 193266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969443 with bad export cookie 1528411688970990316 [ 6308.655277] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.661651] LustreError: 193266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6308.902766] Lustre: server umount lustre-MDT0001 complete [ 6332.828452] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:31:06 (1790969466) [ 6339.140813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.350170] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6339.385704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6339.537949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.871551] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6353.673318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.816350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6353.874738] Lustre: 199561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6359.196178] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6366.697834] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6373.335883] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6380.188543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.403468] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6387.393351] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6387.883581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6396.462587] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6403.773825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.834067] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6404.039629] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.046742] Lustre: Skipped 2 previous similar messages [ 6411.426193] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6419.951232] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6424.033566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6424.037129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.047051] LustreError: Skipped 1 previous similar message [ 6424.048394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6424.081375] Lustre: Skipped 1 previous similar message [ 6424.105343] Lustre: Skipped 8 previous similar messages [ 6429.350956] Lustre: server umount lustre-OST0000 complete [ 6431.146694] LustreError: 201414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.179116] LustreError: 201414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6448.097847] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.108635] LustreError: Skipped 1 previous similar message [ 6451.857339] Lustre: server umount lustre-OST0001 complete [ 6456.801864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6456.814254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.836873] Lustre: Skipped 2 previous similar messages [ 6461.755736] Lustre: server umount lustre-MDT0000 complete [ 6465.650523] LustreError: 199554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969600 with bad export cookie 1528411688970991723 [ 6465.653161] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.660239] LustreError: 199554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6466.097572] Lustre: server umount lustre-MDT0001 complete [ 6491.025859] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:33:44 (1790969624) [ 6505.435928] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6514.510119] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6551.775470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.089466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6552.134234] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6552.237298] Lustre: lustre-MDT0000: new disk, initializing [ 6552.340844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6552.369684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6557.587590] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6570.732403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.874496] Lustre: 207657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6570.937016] LustreError: 207667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6570.960736] LustreError: 207667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6570.976348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6570.980665] Lustre: Skipped 1 previous similar message [ 6571.056315] Lustre: lustre-MDT0001: new disk, initializing [ 6571.224384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6571.229140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6575.894434] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6586.110845] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6592.482466] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.992773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6600.387465] Lustre: lustre-OST0000: new disk, initializing [ 6600.405254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6600.419817] Lustre: 209627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6607.835505] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6608.993507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6609.008367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6609.204266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6617.376355] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6622.232725] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6622.414780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6626.640445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6626.898282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6629.864323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6629.874118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.882342] LustreError: Skipped 1 previous similar message [ 6629.886760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6629.895807] Lustre: Skipped 2 previous similar messages [ 6629.913593] Lustre: Skipped 9 previous similar messages [ 6635.174848] Lustre: server umount lustre-OST0000 complete [ 6653.408125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6653.795555] Lustre: server umount lustre-MDT0000 complete [ 6658.898902] LustreError: 207650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969793 with bad export cookie 1528411688970993277 [ 6658.902953] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6658.910893] LustreError: 207650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6659.474532] Lustre: server umount lustre-MDT0001 complete [ 6670.270973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6676.861116] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6686.965581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.071566] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6701.952747] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6709.151344] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6716.991653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.274466] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6732.516365] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6738.136267] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6740.558716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6746.669445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6746.868356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6749.153243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6755.311505] LustreError: 214580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6755.328223] LustreError: 214580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6755.410984] Lustre: server umount lustre-OST0000 complete [ 6762.395532] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6767.738806] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6796.870235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.693558] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6805.611216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6805.627430] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6810.742177] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6810.761573] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6812.800320] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6815.565792] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6821.353759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6821.359502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.375246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6821.375255] Lustre: Skipped 11 previous similar messages [ 6821.418190] Lustre: Skipped 5 previous similar messages [ 6825.821930] Lustre: server umount lustre-OST0000 complete [ 6836.791095] Lustre: server umount lustre-MDT0000 complete [ 6841.114655] LustreError: 215969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969976 with bad export cookie 1528411688970994166 [ 6841.116344] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6841.129785] LustreError: 215969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6841.689719] Lustre: server umount lustre-MDT0001 complete [ 6852.105157] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:39:45 (1790969985) [ 6861.015697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.974773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6861.984232] Lustre: Skipped 6 previous similar messages [ 6869.149335] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6882.801514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6889.450655] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6900.207300] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6906.296756] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6913.383799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.161209] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6927.331622] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6932.471941] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6932.740946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6938.199350] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6938.377651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6945.224162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6945.437201] Lustre: lustre-OST0001: new disk, initializing [ 6945.441051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6945.447220] Lustre: 221199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6950.997026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6951.009881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6951.098084] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6951.464350] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6961.747220] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6966.610090] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6966.812164] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6971.636492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6971.873125] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6976.996600] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6977.006315] LustreError: Skipped 2 previous similar messages [ 6988.256323] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6988.369494] Lustre: server umount lustre-OST0001 complete [ 6996.664712] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6997.120851] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7003.469466] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7003.936624] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7012.840486] LustreError: 219652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7012.863478] LustreError: 219652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7032.588248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.453584] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7038.476184] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7038.480278] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7038.506449] Lustre: Skipped 1 previous similar message [ 7040.518773] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7049.464554] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7051.567765] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7059.785273] Lustre: server umount lustre-OST0001 complete [ 7079.404329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.418940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7079.424678] Lustre: Skipped 7 previous similar messages [ 7079.430978] Lustre: Skipped 16 previous similar messages [ 7084.639263] Lustre: server umount lustre-OST0000 complete [ 7095.036746] Lustre: server umount lustre-MDT0000 complete [ 7099.116463] LustreError: 218563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970234 with bad export cookie 1528411688970995132 [ 7099.120939] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.127088] LustreError: 218563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7105.656071] Lustre: server umount lustre-MDT0001 complete [ 7135.549517] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:44:29 (1790970269) [ 7143.523126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.944575] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7143.977604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7149.110665] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7158.947810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.089814] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7159.147317] Lustre: 226445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7164.541196] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7174.915244] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7181.698826] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7188.801800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.988200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7196.742332] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7205.104592] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7215.470527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.581346] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7222.521678] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7230.971344] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.813344] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7238.003300] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7238.205611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7242.529226] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7242.759081] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7249.080566] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8964c885b000.ost_server_uuid 50 [ 7250.723056] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8964c885b000.ost_server_uuid in IDLE state after 0 sec [ 7251.952313] Lustre: Permanently deactivating lustre-OST0001 [ 7251.978214] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7255.534909] Lustre: Permanently reactivating lustre-OST0001 [ 7263.727027] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7263.765798] LustreError: Skipped 2 previous similar messages [ 7266.803106] Lustre: server umount lustre-OST0001 complete [ 7278.977668] Lustre: server umount lustre-OST0000 complete [ 7290.861567] Lustre: server umount lustre-MDT0000 complete [ 7295.230223] LustreError: 226437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970430 with bad export cookie 1528411688970996420 [ 7295.237362] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.251256] LustreError: 226437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7295.691873] Lustre: server umount lustre-MDT0001 complete [ 7324.263381] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:47:37 (1790970457) [ 7333.086636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7333.571599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7333.634682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7333.655512] Lustre: Skipped 5 previous similar messages [ 7339.242356] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7349.126050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.331286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7349.374287] Lustre: 233140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7355.042539] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7362.920165] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7370.255609] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7377.801193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.021875] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7378.043210] Lustre: Permanently deactivating lustre-OST0000 [ 7378.058670] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7378.243966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.259507] Lustre: Skipped 10 previous similar messages [ 7385.401940] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7386.661260] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7393.478606] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.007850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7401.145088] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7407.891915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7409.089793] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7418.438544] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7420.074326] Lustre: Permanently reactivating lustre-OST0000 [ 7420.095887] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7420.100904] Lustre: Skipped 2 previous similar messages [ 7427.637964] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7427.638351] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 000000005f5a1ea2, cur 1790970562 deadline 1790970551 last 1790970521 [ 7427.648875] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7427.676928] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7427.754921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7441.849869] Lustre: server umount lustre-OST0001 complete [ 7452.513801] Lustre: server umount lustre-OST0000 complete [ 7464.535381] Lustre: server umount lustre-MDT0000 complete [ 7470.501237] LustreError: 233132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970605 with bad export cookie 1528411688970998065 [ 7470.526261] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.533152] LustreError: 233132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7471.227266] Lustre: server umount lustre-MDT0001 complete [ 7483.968769] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7486.880598] Key type lgssc unregistered [ 7487.311834] LNet: 238019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7487.334273] LNetError: 238019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.358755] LNet: Removed LNI 192.168.204.133@tcp [ 7489.230914] Key type .llcrypt unregistered [ 7489.234790] Key type ._llcrypt unregistered [ 7513.797178] Key type ._llcrypt registered [ 7513.802296] Key type .llcrypt registered [ 7513.904574] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7515.283692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7515.309952] alg: No test for adler32 (adler32-zlib) [ 7516.326653] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7516.604616] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7518.313875] Key type lgssc registered [ 7519.584418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7529.094626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7529.115220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.754927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.451580] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7540.206747] LustreError: 239484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.235898] LustreError: 239484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7544.970420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.101596] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7545.136882] Lustre: 239474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7545.159832] Lustre: Permanently deactivating lustre-MDT0001 [ 7545.178272] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7545.321619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7545.504319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7550.520644] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7551.485608] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7558.491973] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7564.895740] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7573.258808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7573.725771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7575.791364] LustreError: 241327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.804170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7580.600418] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7581.156066] LustreError: 241434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7586.280982] LustreError: 241327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.538255] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7590.374325] LustreError: 241434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.395661] LustreError: 241434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7596.460501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.762341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.170399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7602.174712] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7603.801663] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7612.506746] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7616.740938] Lustre: Permanently reactivating lustre-MDT0001 [ 7616.769353] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7616.780451] Lustre: Skipped 2 previous similar messages [ 7619.499635] Lustre: lustre-MDT0001: Client fd956b86-f2e9-4084-a796-b113054a7a8d (at 192.168.204.33@tcp) reconnecting [ 7625.396212] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7625.411665] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7625.418938] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7625.436217] LustreError: 243206:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7625.437476] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7625.445716] LustreError: 243206:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7625.445941] LustreError: 243206:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.133@tcp: cfg command failed: rc = -108 [ 7625.446121] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7625.446121] [ 7625.446895] LustreError: 239476:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7625.825303] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7627.730685] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7634.935756] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7635.229504] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7640.694841] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7640.891674] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7643.646916] Lustre: Permanently deactivating lustre-MDT0001 [ 7643.662357] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7643.670536] Lustre: Skipped 2 previous similar messages [ 7650.934691] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7657.959479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7657.975071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7658.009032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7659.469955] Lustre: server umount lustre-MDT0000 complete [ 7663.589220] LustreError: 239484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.596612] LustreError: 239484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7663.670830] LustreError: 239466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970798 with bad export cookie 1929726523336717097 [ 7663.676903] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7663.680991] LustreError: 239466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7664.114770] Lustre: server umount lustre-MDT0001 complete [ 7670.618463] Lustre: server umount lustre-OST0000 complete [ 7684.554956] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:53:38 (1790970818) [ 7696.961796] Lustre: server umount lustre-OST0001 complete [ 7703.983289] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 7714.843641] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7755.857550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7756.111625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7756.119158] Lustre: Skipped 2 previous similar messages [ 7756.137262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7756.205719] Lustre: lustre-MDT0000: new disk, initializing [ 7756.284733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.321521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7762.232382] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7773.722380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.881458] Lustre: 247742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7773.941747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7773.949781] Lustre: Skipped 1 previous similar message [ 7774.071505] Lustre: lustre-MDT0001: new disk, initializing [ 7774.247776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7774.289183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7774.302488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7779.799440] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7791.585527] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7798.766606] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7809.508388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7809.933649] Lustre: lustre-OST0000: new disk, initializing [ 7809.941422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7809.951989] Lustre: 249712:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7810.078138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.273783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7816.290530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7816.453663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7817.691471] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7827.626438] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7830.773265] LustreError: 249114:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7840.813301] LustreError: 249114:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7840.853167] LustreError: 247750:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7841.360648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7841.663933] Lustre: lustre-OST0001: new disk, initializing [ 7841.667785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7841.684085] Lustre: 250828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7848.701440] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7849.033943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7849.050402] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7849.166642] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7850.856150] LustreError: 247750:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7859.630409] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7861.333184] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7864.810274] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.811647] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7864.815317] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7864.815324] Lustre: Skipped 4 previous similar messages [ 7864.833816] Lustre: Skipped 4 previous similar messages [ 7864.865221] LustreError: Skipped 1 previous similar message [ 7869.855602] Lustre: server umount lustre-OST0001 complete [ 7869.932076] LustreError: 251209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.957746] LustreError: 251209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7888.354042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7888.356628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.372120] LustreError: Skipped 1 previous similar message [ 7888.388949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7888.395806] Lustre: Skipped 2 previous similar messages [ 7895.525371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7895.540788] Lustre: Skipped 3 previous similar messages [ 7902.179169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7902.477930] Lustre: server umount lustre-OST0000 complete [ 7907.297499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7907.312060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7907.333892] Lustre: Skipped 1 previous similar message [ 7907.359210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7907.366724] Lustre: Skipped 2 previous similar messages [ 7913.456442] Lustre: server umount lustre-MDT0000 complete [ 7916.031532] LustreError: 247755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.059145] LustreError: 247755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7917.674995] LustreError: 247735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971052 with bad export cookie 1929726523336719694 [ 7917.681533] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7917.686103] LustreError: 247735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7931.872269] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7932.148630] Lustre: server umount lustre-MDT0001 complete [ 7945.037893] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7948.442809] Key type lgssc unregistered [ 7948.800514] LNet: 252867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7948.813358] LNetError: 252867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7948.826506] LNet: Removed LNI 192.168.204.133@tcp [ 7949.917074] Key type .llcrypt unregistered [ 7949.922466] Key type ._llcrypt unregistered [ 7993.401820] Key type ._llcrypt registered [ 7993.406398] Key type .llcrypt registered [ 7993.546411] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7995.114976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7995.193255] alg: No test for adler32 (adler32-zlib) [ 7996.317462] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7996.607846] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7998.296205] Key type lgssc registered [ 7999.568477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8009.820265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8009.841475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8011.282602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8011.323398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8011.536791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.761129] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8029.514709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8029.677353] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8029.734371] Lustre: 255078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8030.175992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8035.174581] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8045.707367] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8053.245233] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8062.170514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.399906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8062.618750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.419790] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8071.892512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8080.183296] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8138.213465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.224728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.246293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8138.742349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.748773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.777936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.757606] Lustre: server umount lustre-OST0000 complete [ 8146.336719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8160.534675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.115196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.644558] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8177.511246] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8178.491959] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8178.515980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8178.517820] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8178.559289] Lustre: Skipped 1 previous similar message [ 8219.618198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.637595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8219.641507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8219.652314] Lustre: Skipped 1 previous similar message [ 8219.673651] Lustre: Skipped 1 previous similar message [ 8223.922744] Lustre: server umount lustre-OST0000 complete [ 8229.857959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8229.863968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8229.864206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8229.876815] Lustre: Skipped 1 previous similar message [ 8234.346504] Lustre: server umount lustre-MDT0000 complete [ 8238.547751] LustreError: 255071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971373 with bad export cookie 7296682452962617435 [ 8238.562670] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8238.571096] LustreError: 255071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8238.879784] Lustre: server umount lustre-MDT0001 complete [ 8252.293627] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8256.161963] Key type lgssc unregistered [ 8256.560718] LNet: 260512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8256.568025] LNetError: 260512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8256.607766] LNet: Removed LNI 192.168.204.133@tcp [ 8257.804269] Key type .llcrypt unregistered [ 8257.813334] Key type ._llcrypt unregistered [ 8287.275579] Key type ._llcrypt registered [ 8287.280743] Key type .llcrypt registered [ 8287.422377] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8289.163031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8289.234106] alg: No test for adler32 (adler32-zlib) [ 8290.388663] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8290.702461] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8292.464235] Key type lgssc registered [ 8294.149745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.714297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.737716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.328652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8312.566703] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8316.903119] LustreError: 261926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8316.942366] LustreError: 261926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8322.034934] LustreError: 261927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.606380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.361647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8332.405169] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8343.697695] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8351.749640] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8360.707522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8361.126280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8369.505722] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8376.500423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8378.752476] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8406.502054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8406.519821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8406.557566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8407.527096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8407.539323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8407.541067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8407.578909] Lustre: Skipped 1 previous similar message [ 8412.383845] Lustre: server umount lustre-OST0000 complete [ 8417.249711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8417.258247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.281015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8422.784405] Lustre: server umount lustre-MDT0000 complete [ 8422.893050] LustreError: 261922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8422.921331] LustreError: 261922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8426.890717] LustreError: 262662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971561 with bad export cookie 5545909233062377309 [ 8426.892376] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8426.913614] LustreError: 262662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8427.319532] Lustre: server umount lustre-MDT0001 complete [ 8439.737607] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8442.814693] Key type lgssc unregistered [ 8443.160198] LNet: 266234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8443.176517] LNetError: 266234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8443.200865] LNet: Removed LNI 192.168.204.133@tcp [ 8444.171351] Key type .llcrypt unregistered [ 8444.173839] Key type ._llcrypt unregistered [ 8466.032689] Key type ._llcrypt registered [ 8466.037902] Key type .llcrypt registered [ 8466.167895] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8467.577342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8467.689232] alg: No test for adler32 (adler32-zlib) [ 8468.976498] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8469.261346] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8471.017885] Key type lgssc registered [ 8472.773432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8482.363600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8482.408410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.365401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8491.179548] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8494.565050] LustreError: 267453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.607609] LustreError: 267453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8499.688588] LustreError: 267454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.031392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.662389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8511.136635] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8522.212772] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.780740] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8538.014758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8538.273198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8545.219635] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8554.230094] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8554.674765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8570.344797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.354090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.399916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.417476] Lustre: Skipped 1 previous similar message [ 8574.775389] Lustre: server umount lustre-OST0000 complete [ 8580.579218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.599585] Lustre: Skipped 1 previous similar message [ 8580.606339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8585.699549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8585.718150] Lustre: Skipped 2 previous similar messages [ 8586.885070] Lustre: server umount lustre-MDT0000 complete [ 8590.819159] LustreError: 267453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.843281] LustreError: 267453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8592.492168] LustreError: 267434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971727 with bad export cookie 3362207390002500106 [ 8592.503931] LustreError: 267434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8592.506387] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8593.477712] Lustre: server umount lustre-MDT0001 complete [ 8606.138093] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8609.470146] Key type lgssc unregistered [ 8609.827803] LNet: 271135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8609.837752] LNetError: 271135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8609.859137] LNet: Removed LNI 192.168.204.133@tcp [ 8611.079912] Key type .llcrypt unregistered [ 8611.084958] Key type ._llcrypt unregistered [ 8637.974931] Key type ._llcrypt registered [ 8637.978745] Key type .llcrypt registered [ 8638.102922] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8639.462086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8639.491129] alg: No test for adler32 (adler32-zlib) [ 8640.516182] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8640.803934] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8642.496209] Key type lgssc registered [ 8643.835799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8652.301201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8652.323479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.849874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.034558] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8663.013589] LustreError: 272547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.030645] LustreError: 272547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8668.007257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.134801] LustreError: 272546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.608145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8673.660499] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8681.751444] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8687.734913] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8694.889797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8695.369919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8698.434938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8702.851209] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8710.514733] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8739.299447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8739.304534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.316089] LustreError: Skipped 1 previous similar message [ 8739.331925] Lustre: Skipped 1 previous similar message [ 8739.332487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8739.339229] Lustre: Skipped 1 previous similar message [ 8744.453458] Lustre: server umount lustre-OST0000 complete [ 8749.539809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.542089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.554163] Lustre: Skipped 1 previous similar message [ 8749.568486] Lustre: Skipped 1 previous similar message [ 8754.657323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8754.667154] Lustre: Skipped 1 previous similar message [ 8754.970099] Lustre: server umount lustre-MDT0000 complete [ 8758.785506] LustreError: 276103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971893 with bad export cookie 9948843664537697477 [ 8758.787557] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8758.796472] LustreError: 276103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8759.299242] Lustre: server umount lustre-MDT0001 complete [ 8772.455549] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8776.447658] Key type lgssc unregistered [ 8776.803179] LNet: 276853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8776.814411] LNetError: 276853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8776.835071] LNet: Removed LNI 192.168.204.133@tcp [ 8778.115154] Key type .llcrypt unregistered [ 8778.123195] Key type ._llcrypt unregistered [ 8795.897420] Key type ._llcrypt registered [ 8795.899814] Key type .llcrypt registered [ 8796.057935] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8797.973706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8798.034639] alg: No test for adler32 (adler32-zlib) [ 8799.236531] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8799.509779] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8801.224178] Key type lgssc registered [ 8802.675608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8811.074526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8811.116161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.610856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.699349] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8822.755887] LustreError: 278083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8822.790547] LustreError: 278083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8827.273570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8827.754379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8832.928884] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8840.392001] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8847.172624] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8854.348271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.830194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8857.910436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8861.868465] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8869.435803] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8883.688880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8883.698314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.709487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8887.781534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.791330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8887.818121] Lustre: Skipped 1 previous similar message [ 8889.433278] Lustre: server umount lustre-OST0000 complete [ 8894.434421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8894.437884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.461619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8898.018333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8898.023372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8899.599132] Lustre: server umount lustre-MDT0000 complete [ 8903.141926] LustreError: 278085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8903.156288] LustreError: 278085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8903.650532] LustreError: 278051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972038 with bad export cookie 3918104563258592846 [ 8903.678789] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8904.068102] Lustre: server umount lustre-MDT0001 complete [ 8915.969375] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8919.063648] Key type lgssc unregistered [ 8919.421773] LNet: 281782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8919.428575] LNetError: 281782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8919.448623] LNet: Removed LNI 192.168.204.133@tcp [ 8920.589110] Key type .llcrypt unregistered [ 8920.591773] Key type ._llcrypt unregistered [ 8932.408954] Key type ._llcrypt registered [ 8932.412413] Key type .llcrypt registered [ 8932.575724] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8951.535504] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 8966.286705] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8967.341081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8967.525074] alg: No test for adler32 (adler32-zlib) [ 8968.612973] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8968.893467] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8970.744653] Key type lgssc registered [ 8971.830752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9003.714682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9003.742315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.019735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9005.092325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9005.207603] Lustre: lustre-MDT0000: new disk, initializing [ 9005.308755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9005.346060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9009.407791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9018.924491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9019.067621] Lustre: 285330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9019.133885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9019.138598] Lustre: Skipped 1 previous similar message [ 9019.229695] Lustre: lustre-MDT0001: new disk, initializing [ 9019.335924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9019.374351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9019.379959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9023.278641] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9032.782660] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9038.472564] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9045.314029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9045.674110] Lustre: lustre-OST0000: new disk, initializing [ 9045.682059] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9045.692448] Lustre: 287302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9045.791691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.097537] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9054.315652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9054.331045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9054.500421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9063.525772] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9068.129955] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9068.426585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9073.199548] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9073.426871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9080.302289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9080.306938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.311988] LustreError: Skipped 1 previous similar message [ 9080.312051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9080.347702] Lustre: Skipped 1 previous similar message [ 9081.856856] Lustre: server umount lustre-OST0000 complete [ 9085.922066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9085.931123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.946946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9085.958846] Lustre: Skipped 1 previous similar message [ 9090.531509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9090.532638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9090.568398] Lustre: Skipped 1 previous similar message [ 9091.830274] Lustre: server umount lustre-MDT0000 complete [ 9095.926237] LustreError: 285325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972230 with bad export cookie 11887846338914202301 [ 9095.930600] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.942334] LustreError: 285325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9096.409840] Lustre: server umount lustre-MDT0001 complete [ 9106.014633] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:17:19 (1790972239) [ 9112.178358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.551541] LustreError: 289713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.573916] LustreError: 289713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9112.675297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9117.013838] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9117.669962] LustreError: 289714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.790169] LustreError: 289713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9125.967639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9126.343747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9130.499660] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9137.855350] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9143.535247] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9150.826930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9151.424594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.464612] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9166.173959] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9178.086714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.086961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9178.114389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9182.030728] Lustre: server umount lustre-OST0000 complete [ 9187.809646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9187.821390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.839237] Lustre: Skipped 1 previous similar message [ 9187.845672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9187.852029] Lustre: Skipped 1 previous similar message [ 9192.726352] Lustre: server umount lustre-MDT0000 complete [ 9193.442653] LustreError: 290954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9193.469589] LustreError: 290954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9197.153418] LustreError: 289695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972332 with bad export cookie 11887846338914203183 [ 9197.154548] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9197.169090] LustreError: 289695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9197.887283] Lustre: server umount lustre-MDT0001 complete [ 9211.581347] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 9214.231233] Key type lgssc unregistered [ 9214.538839] LNet: 293249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9214.554056] LNetError: 293249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9214.583557] LNet: Removed LNI 192.168.204.133@tcp [ 9215.407955] Key type .llcrypt unregistered [ 9215.410469] Key type ._llcrypt unregistered [ 9242.495471] Key type ._llcrypt registered [ 9242.503444] Key type .llcrypt registered [ 9242.667478] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 9258.631732] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9259.525318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9259.556094] alg: No test for adler32 (adler32-zlib) [ 9260.637619] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 9260.915365] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 9262.624199] Key type lgssc registered [ 9264.005873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9315.101722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9315.155619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.443224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9316.521652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9316.675345] Lustre: lustre-MDT0000: new disk, initializing [ 9316.818431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9316.843562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9321.922274] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9333.789846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.887473] Lustre: 297072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9333.974969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9333.981564] Lustre: Skipped 1 previous similar message [ 9334.061904] Lustre: lustre-MDT0001: new disk, initializing [ 9334.156213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9334.199230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9334.207788] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9338.966588] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9348.999468] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9355.085552] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9363.217032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9363.439727] Lustre: lustre-OST2710: new disk, initializing [ 9363.444188] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9363.449867] Lustre: 299041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9363.529879] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9370.644599] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9372.273366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9372.291547] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9372.434393] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9381.210784] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9389.519792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9389.802867] Lustre: lustre-OST03e8: new disk, initializing [ 9389.814795] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9389.823527] Lustre: 300106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9389.924720] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9396.350876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9396.359847] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9396.478603] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9398.688410] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9412.120986] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9424.561361] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9425.122992] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9430.903149] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9431.436777] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9436.779805] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9436.959227] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9442.452822] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9442.641983] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9447.921570] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.926571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.944410] Lustre: Skipped 1 previous similar message [ 9447.962185] Lustre: Skipped 3 previous similar messages [ 9449.628792] Lustre: server umount lustre-MDT0000 complete [ 9454.217313] LustreError: 297065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972589 with bad export cookie 7356140220958722546 [ 9454.226028] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9454.239455] LustreError: 297065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9454.704907] Lustre: server umount lustre-MDT0001 complete [ 9469.298562] Lustre: server umount lustre-OST2710 complete [ 9484.086195] Lustre: server umount lustre-OST03e8 complete [ 9506.705362] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 9519.148987] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9564.291919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.576487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9564.631693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9564.726383] Lustre: lustre-MDT0000: new disk, initializing [ 9564.866505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9564.897576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9570.422362] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9583.339521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.554354] Lustre: 305364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9583.667306] LustreError: 305375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.684927] LustreError: 305375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9583.710758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9583.719124] Lustre: Skipped 1 previous similar message [ 9583.784330] Lustre: lustre-MDT0001: new disk, initializing [ 9583.912758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9583.963206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9583.986910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9589.030990] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9599.646841] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9605.761446] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9613.738135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9614.156281] Lustre: lustre-OST0000: new disk, initializing [ 9614.171505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9614.185178] Lustre: 307334:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9614.367609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.633731] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9623.139733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9623.148229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9623.305660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9632.823505] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9637.818420] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9638.029122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9643.901543] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9644.167994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9649.133633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9649.146262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.150986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9649.161610] Lustre: Skipped 3 previous similar messages [ 9649.176267] Lustre: Skipped 1 previous similar message [ 9652.868854] Lustre: server umount lustre-OST0000 complete [ 9659.361907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.368491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.394635] Lustre: Skipped 1 previous similar message [ 9659.415738] Lustre: Skipped 1 previous similar message [ 9664.113593] Lustre: server umount lustre-MDT0000 complete [ 9664.484221] LustreError: 305375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9664.507549] LustreError: 305375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9668.878712] LustreError: 305355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972803 with bad export cookie 7356140220958724219 [ 9668.881397] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9668.894048] LustreError: 305355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9669.526368] Lustre: server umount lustre-MDT0001 complete [ 9680.499527] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:26:53 (1790972813) [ 9682.247498] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9684.509663] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:26:57 (1790972817) [ 9688.375924] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9715.563895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9715.827330] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9715.846686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9715.966964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.203887] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9730.591555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.681691] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9730.713979] Lustre: 311193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9736.539654] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9759.401038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9759.556835] Lustre: Denying initial registration attempt from nid 192.168.204.133@tcp, specified as failover [ 9759.563464] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9759.572663] LustreError: 312647:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9759.578362] LustreError: 312647:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9759.583534] LustreError: 312647:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9759.658215] Lustre: server umount lustre-OST0000 complete [ 9759.669689] LustreError: 312647:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9772.002805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9772.016749] LustreError: Skipped 1 previous similar message [ 9772.025871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9772.053324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.681368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.688292] Lustre: Skipped 3 previous similar messages [ 9784.288164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9784.624071] Lustre: server umount lustre-MDT0000 complete [ 9784.802618] LustreError: 312649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9784.820491] LustreError: 312649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9789.815449] LustreError: 311184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972924 with bad export cookie 7356140220958725108 [ 9789.834978] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9790.462816] Lustre: server umount lustre-MDT0001 complete [ 9802.839217] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 9806.645789] Key type lgssc unregistered [ 9806.981853] LNet: 313771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9806.991064] LNetError: 313771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9807.013774] LNet: Removed LNI 192.168.204.133@tcp [ 9807.963953] Key type .llcrypt unregistered [ 9807.970136] Key type ._llcrypt unregistered [ 9821.741383] Key type ._llcrypt registered [ 9821.743062] Key type .llcrypt registered [ 9821.892361] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9822.866152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9822.939745] alg: No test for adler32 (adler32-zlib) [ 9824.053963] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 9824.286457] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 9825.936231] Key type lgssc registered [ 9827.306881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9885.869434] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9895.626214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9895.658466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.975438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9897.038192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9897.310320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9902.854611] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9913.425666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.621199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9913.678773] Lustre: 316534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9914.034251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9919.071677] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9928.399167] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9936.021827] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9958.704434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9958.925617] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9959.143675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9966.757495] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9974.915463] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9982.435466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9982.448638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.468842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9988.086554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9988.102171] Lustre: Skipped 2 previous similar messages [ 9992.160296] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9992.583401] Lustre: server umount lustre-OST0000 complete [ 9996.259909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9996.266607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.289282] Lustre: Skipped 1 previous similar message [ 9996.295910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9998.314275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9998.314774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9998.361536] Lustre: Skipped 1 previous similar message [10002.674372] Lustre: server umount lustre-MDT0000 complete [10006.730879] LustreError: 316526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973141 with bad export cookie 15005534840445904329 [10006.743847] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.749893] LustreError: 316526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10007.056924] Lustre: server umount lustre-MDT0001 complete [10020.993526] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10023.935216] Key type lgssc unregistered [10024.286884] LNet: 320109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10024.300230] LNetError: 320109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10024.344723] LNet: Removed LNI 192.168.204.133@tcp [10025.260729] Key type .llcrypt unregistered [10025.263091] Key type ._llcrypt unregistered [10052.385577] Key type ._llcrypt registered [10052.388568] Key type .llcrypt registered [10052.534446] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10053.452619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10053.613504] alg: No test for adler32 (adler32-zlib) [10054.615225] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [10054.893750] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [10056.576164] Key type lgssc registered [10057.709300] Lustre: Echo OBD driver; http://www.lustre.org/ [10065.022983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10065.035453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10066.444574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10071.366665] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10076.646480] LustreError: 321522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10076.660447] LustreError: 321522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10079.078355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10079.437781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10083.712452] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10091.498731] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10097.617548] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10108.481860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10108.879548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10115.453104] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10123.891561] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10135.523513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.526606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10135.561042] Lustre: Skipped 2 previous similar messages [10135.604152] Lustre: Skipped 2 previous similar messages [10137.035854] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [10139.067023] Lustre: server umount lustre-MDT0000 complete [10140.642058] LustreError: 321522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10140.678110] LustreError: 321522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10142.147206] LustreError: 321518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.261299] LustreError: 322257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973278 with bad export cookie 14102870403738914900 [10143.264171] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10143.284895] LustreError: 322257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10145.761045] LustreError: 321517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10145.765148] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.846239] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10148.773506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10148.778933] Lustre: Skipped 1 previous similar message [10149.834862] Lustre: server umount lustre-MDT0001 complete [10158.959131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10168.745589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.002339] LustreError: 323356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.025254] LustreError: 323356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10189.096363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10189.147402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10194.411462] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10198.500237] LustreError: 325118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.531859] LustreError: 325118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10202.872067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10203.256562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10204.271727] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10208.284517] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10215.924674] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10217.438814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10223.078680] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10225.067356] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10230.248721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10230.278781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10230.284090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10230.288915] Lustre: Skipped 1 previous similar message [10240.487539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10240.496490] Lustre: Skipped 3 previous similar messages [10244.064183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10244.270646] Lustre: server umount lustre-OST0000 complete [10249.185563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10249.194943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10249.217512] Lustre: Skipped 1 previous similar message [10254.430963] Lustre: server umount lustre-MDT0000 complete [10255.847443] LustreError: 325123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.865614] LustreError: 325123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10258.373635] LustreError: 322257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973393 with bad export cookie 14102870403738922572 [10258.380716] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10258.388847] LustreError: 322257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10258.765834] Lustre: server umount lustre-MDT0001 complete [10272.701640] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10276.347490] Key type lgssc unregistered [10276.753568] LNet: 327795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10276.759430] LNetError: 327795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10276.794068] LNet: Removed LNI 192.168.204.133@tcp [10278.140276] Key type .llcrypt unregistered [10278.141829] Key type ._llcrypt unregistered [10305.189563] Key type ._llcrypt registered [10305.191392] Key type .llcrypt registered [10305.379537] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10306.743696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10306.765892] alg: No test for adler32 (adler32-zlib) [10307.971681] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [10308.219388] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [10309.867396] Key type lgssc registered [10310.923213] Lustre: Echo OBD driver; http://www.lustre.org/ [10320.419839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10320.444548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.162699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10328.095812] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10332.650491] LustreError: 329213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.680650] LustreError: 329213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10337.773266] LustreError: 329214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.748246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.178114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10344.207363] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10351.934396] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10358.317071] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10361.756186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10361.777529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10361.785561] Lustre: Skipped 1 previous similar message [10364.833703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10364.842027] Lustre: Skipped 1 previous similar message [10370.021073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10370.032423] Lustre: Skipped 1 previous similar message [10375.143608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.153133] Lustre: Skipped 1 previous similar message [10376.160238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10376.540676] Lustre: server umount lustre-MDT0000 complete [10380.258874] LustreError: 329214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10380.288838] LustreError: 329214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10381.184114] LustreError: 329194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973516 with bad export cookie 12084095276559820470 [10381.205659] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10381.599547] Lustre: server umount lustre-MDT0001 complete [10399.909977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10400.408973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10400.485957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10400.852838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.594461] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10416.601291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10416.711031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10416.748807] Lustre: 331801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10417.098413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10421.729531] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10430.316517] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10436.592940] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10447.221174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.430747] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10447.443684] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10447.453926] LustreError: 333688:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10447.463860] LustreError: 333688:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10447.467404] LustreError: 333688:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10447.563363] Lustre: server umount lustre-OST0000 complete [10447.567094] LustreError: 333688:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10459.318128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10459.603250] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10459.857446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10466.372225] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10467.457628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10473.965827] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10481.830793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10481.893090] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10481.962440] Lustre: lustre-OST0001: new disk, initializing [10481.965648] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10481.973387] Lustre: 335230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10482.040677] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10488.445646] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10491.511475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10491.535118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10491.644179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10495.815290] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10502.126490] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10502.127120] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10502.134906] LustreError: Skipped 1 previous similar message [10502.141256] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10502.145118] Lustre: Skipped 2 previous similar messages [10502.161878] Lustre: Skipped 1 previous similar message [10504.626693] Lustre: server umount lustre-OST0001 complete [10507.242045] LustreError: 334155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10507.263927] LustreError: 334155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10508.771521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10508.776050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.355870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10512.357461] LustreError: 334156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10512.363310] Lustre: Skipped 1 previous similar message [10512.374846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.396104] LustreError: 334156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10522.611038] LustreError: 334156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10522.650051] LustreError: 334156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10523.104403] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10523.311551] Lustre: server umount lustre-OST0000 complete [10529.765897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10529.776049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10529.793378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10529.801040] Lustre: Skipped 5 previous similar messages [10534.419592] Lustre: server umount lustre-MDT0000 complete [10538.827582] LustreError: 331795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973673 with bad export cookie 12084095276559821030 [10538.831819] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10538.843819] LustreError: 331795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10539.208422] Lustre: server umount lustre-MDT0001 complete [10555.455683] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10559.987567] Key type lgssc unregistered [10560.379268] LNet: 337190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10560.389711] LNetError: 337190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10560.423134] LNet: Removed LNI 192.168.204.133@tcp [10562.102718] Key type .llcrypt unregistered [10562.107399] Key type ._llcrypt unregistered [10645.999863] Key type ._llcrypt registered [10646.003872] Key type .llcrypt registered [10646.101337] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [10663.551756] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10665.521806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10665.666856] alg: No test for adler32 (adler32-zlib) [10667.191255] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [10667.540702] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [10669.304184] Key type lgssc registered [10672.033319] Lustre: Echo OBD driver; http://www.lustre.org/ [10719.211912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10719.232825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10720.450822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10720.490856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10720.577705] Lustre: lustre-MDT0000: new disk, initializing [10720.651684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10720.679914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10725.943586] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10736.244132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.379327] Lustre: 342127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10736.449451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10736.456511] Lustre: Skipped 1 previous similar message [10736.564345] Lustre: lustre-MDT0001: new disk, initializing [10736.720104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10736.763886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10736.785911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10741.539323] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10751.190665] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10757.548274] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10765.012465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10765.489823] Lustre: lustre-OST0000: new disk, initializing [10765.499503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10765.512687] Lustre: 344096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10765.625744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10771.809749] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10774.090207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10774.098661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10774.238729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10781.001968] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10785.869212] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10786.273680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10791.551497] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10791.814897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10794.977795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10794.989332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10795.029299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10795.040430] Lustre: Skipped 1 previous similar message [10798.074097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10798.086360] Lustre: Skipped 1 previous similar message [10803.176151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10803.193691] Lustre: Skipped 1 previous similar message [10808.290871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10808.291809] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [10808.308942] Lustre: Skipped 1 previous similar message [10808.595764] Lustre: server umount lustre-OST0000 complete [10813.413862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.415107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10813.422915] Lustre: Skipped 1 previous similar message [10813.437074] LustreError: Skipped 1 previous similar message [10813.440393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10818.869399] Lustre: server umount lustre-MDT0000 complete [10823.580183] LustreError: 342120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973958 with bad export cookie 5112326720158260444 [10823.587728] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10823.595243] LustreError: 342120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10824.115994] Lustre: server umount lustre-MDT0001 complete [10835.892537] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:46:09 (1790973969) [10843.012656] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:46:16 (1790973976) [10851.750683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.425324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10857.449280] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.491195] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10857.560893] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10862.561735] LustreError: 346851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10867.689743] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10868.010476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.597703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10874.543630] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10885.240087] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10892.643978] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10901.629035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10902.039893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10909.773140] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10919.431643] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10931.851833] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10934.112391] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10936.226690] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10938.515656] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10940.273610] Lustre: Failing over lustre-MDT0000 [10940.397249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10940.406991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10940.435229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10940.440963] Lustre: Skipped 3 previous similar messages [10940.888242] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.596736] Lustre: server umount lustre-MDT0000 complete [10944.491196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10944.506168] Lustre: Skipped 1 previous similar message [10945.960837] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.987463] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10954.736699] LustreError: 348095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.755459] LustreError: 348095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10960.800201] Lustre: 340348:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790974079/real 1790974079] req@0000000008c2e8a7 x1877972145482624/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790974095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10960.819704] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.408966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.081322] LustreError: 340345:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006fe03160 x1877972145487616/t0(0) o250->MGC192.168.204.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10970.436722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10970.510936] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10972.605973] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10975.738174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10975.807777] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10975.808341] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10975.873331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10987.739686] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.513931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10993.772807] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10995.932911] Lustre: Failing over lustre-MDT0000 [10996.197041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10996.205971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.223915] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.246884] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10996.273757] Lustre: server umount lustre-MDT0000 complete [11012.585314] Lustre: 340347:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790974131/real 1790974131] req@00000000d543e48a x1877972145503616/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790974147 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11012.622218] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11017.635797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11022.819473] LustreError: 340345:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dff31891 x1877972145509504/t0(0) o250->MGC192.168.204.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11023.323923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11023.426587] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11023.782918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11028.465712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11028.478397] Lustre: Skipped 2 previous similar messages [11028.510694] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11028.574220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11028.578408] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11038.282410] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11040.138893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11048.928785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11048.931688] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.953205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11048.961843] Lustre: Skipped 3 previous similar messages [11048.976472] Lustre: Skipped 2 previous similar messages [11051.111830] Lustre: server umount lustre-MDT0000 complete [11054.051509] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11054.073927] LustreError: 346852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [11054.429276] LustreError: 346832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974189 with bad export cookie 5112326720158263153 [11054.437758] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11054.458455] LustreError: 346832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11054.751912] Lustre: server umount lustre-MDT0001 complete [11067.368270] Lustre: server umount lustre-OST0000 complete [11082.687094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.156597] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11096.999269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11097.305854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11097.313583] Lustre: Skipped 1 previous similar message [11102.182264] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11111.695725] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11118.477821] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11126.258964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11128.641754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11133.597895] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11140.950608] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11152.873573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11152.882710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.901867] Lustre: Skipped 1 previous similar message [11152.919636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11156.835732] Lustre: server umount lustre-OST0000 complete [11175.392149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11175.591892] Lustre: server umount lustre-MDT0000 complete [11178.979862] LustreError: 354229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11178.994544] LustreError: 354229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11179.125945] LustreError: 353202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974314 with bad export cookie 5112326720158263902 [11179.128375] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.132687] LustreError: 353202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11179.442954] Lustre: server umount lustre-MDT0001 complete [11190.656723] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [11193.083907] Key type lgssc unregistered [11193.365423] LNet: 356755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11193.382985] LNetError: 356755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11193.414261] LNet: Removed LNI 192.168.204.133@tcp [11194.275334] Key type .llcrypt unregistered [11194.276968] Key type ._llcrypt unregistered [11222.957315] Key type ._llcrypt registered [11222.959643] Key type .llcrypt registered [11223.115660] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [11225.183894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11225.248209] alg: No test for adler32 (adler32-zlib) [11226.473541] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [11226.854490] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [11228.576450] Key type lgssc registered [11230.128856] Lustre: Echo OBD driver; http://www.lustre.org/ [11238.291993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11238.312932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11239.863192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11244.341383] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11249.127536] LustreError: 358275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11249.148954] LustreError: 358275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11252.006826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.313502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11256.445585] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11262.858724] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11267.866615] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11274.488769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11274.913279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11282.817850] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11284.147431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11290.317254] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11302.858974] Lustre: server umount lustre-MDT0000 complete [11303.405696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11303.425489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11303.471176] LustreError: 358276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.944926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11304.946950] LustreError: 358275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.994200] LustreError: 358275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11308.460558] LustreError: 358257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974443 with bad export cookie 15249888516376679284 [11308.485926] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11308.887816] Lustre: server umount lustre-MDT0001 complete [11323.176744] Lustre: server umount lustre-OST0000 complete [11336.579635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11337.065825] LustreError: 361986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11337.126283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11341.883212] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11342.306592] LustreError: 361987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11350.670082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11351.135834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11355.421519] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11363.162534] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11369.349928] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11376.118504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11376.568596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11382.794838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11382.969493] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11390.885749] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11403.248984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.263504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11403.277023] Lustre: Skipped 3 previous similar messages [11403.289911] Lustre: Skipped 2 previous similar messages [11408.357267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11408.366382] Lustre: Skipped 2 previous similar messages [11409.172480] Lustre: server umount lustre-MDT0000 complete [11412.967122] LustreError: 361966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974548 with bad export cookie 15249888516376680404 [11412.970554] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11412.982549] LustreError: 361966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11413.348529] Lustre: server umount lustre-MDT0001 complete [11427.895202] Lustre: server umount lustre-OST0000 complete [11442.944887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11443.428000] LustreError: 365896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.448190] LustreError: 365896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11443.545336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11448.609473] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11456.788180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.148807] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11469.733871] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11476.156367] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11484.057685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11484.427311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11484.431638] Lustre: Skipped 1 previous similar message [11488.500376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11491.855563] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11500.856809] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11508.195186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.204239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11508.209994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11508.215277] Lustre: Skipped 1 previous similar message [11511.900159] Lustre: server umount lustre-OST0000 complete [11518.435489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11518.436341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.453466] Lustre: Skipped 1 previous similar message [11518.469147] Lustre: Skipped 1 previous similar message [11521.985546] Lustre: server umount lustre-MDT0000 complete [11523.555926] LustreError: 368171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.580997] LustreError: 368171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11525.805938] LustreError: 367739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974660 with bad export cookie 15249888516376681426 [11525.814958] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.821562] LustreError: 367739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11526.115330] Lustre: server umount lustre-MDT0001 complete [11536.981956] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [11539.765130] Key type lgssc unregistered [11540.100890] LNet: 369426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11540.116745] LNetError: 369426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11540.147612] LNet: Removed LNI 192.168.204.133@tcp [11541.213596] Key type .llcrypt unregistered [11541.216791] Key type ._llcrypt unregistered [11566.033531] Key type ._llcrypt registered [11566.036581] Key type .llcrypt registered [11566.163647] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [11567.550852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11567.640708] alg: No test for adler32 (adler32-zlib) [11568.647301] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [11568.899275] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [11570.666111] Key type lgssc registered [11571.778683] Lustre: Echo OBD driver; http://www.lustre.org/ [11579.684661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11579.701775] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.712815] LustreError: 370863:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11579.724521] LustreError: 370863:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11579.733179] LustreError: 370863:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11579.739494] LustreError: 370863:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11579.749983] LustreError: 370863:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11586.205268] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11586.211185] LustreError: 371261:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11586.228884] LustreError: 371261:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11586.238743] LustreError: 371261:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11586.246778] LustreError: 371261:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11586.258837] LustreError: 371261:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11609.445662] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [11612.856817] Key type lgssc unregistered [11613.201558] LNet: 372264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11613.213325] LNetError: 372264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11613.231616] LNet: Removed LNI 192.168.204.133@tcp [11614.074871] Key type .llcrypt unregistered [11614.079114] Key type ._llcrypt unregistered [11636.780468] Key type ._llcrypt registered [11636.782241] Key type .llcrypt registered [11636.904656] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [11656.395268] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [11657.598177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11657.627661] alg: No test for adler32 (adler32-zlib) [11658.832779] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [11659.115297] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [11660.777546] Key type lgssc registered [11662.604870] Lustre: Echo OBD driver; http://www.lustre.org/ [11709.163559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11709.183243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.383599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11710.419344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11710.502428] Lustre: lustre-MDT0000: new disk, initializing [11710.606952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11710.637906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11715.049582] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11726.239877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11726.409318] Lustre: 375425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11726.522381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11726.532569] Lustre: Skipped 1 previous similar message [11726.692689] Lustre: lustre-MDT0001: new disk, initializing [11726.797388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11726.864622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11726.893425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11731.607413] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11741.310145] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11747.464875] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11754.423266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11754.684582] Lustre: lustre-OST0000: new disk, initializing [11754.689713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11754.700718] Lustre: 377406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11754.808165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11761.783493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11761.798704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11761.823610] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11761.916750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11772.269901] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11777.219887] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11777.550239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11782.785252] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11783.026233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11787.745595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11787.759964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.764495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11787.785452] Lustre: Skipped 1 previous similar message [11791.756391] Lustre: server umount lustre-OST0000 complete [11797.998274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.028198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11798.035191] Lustre: Skipped 2 previous similar messages [11802.525668] Lustre: server umount lustre-MDT0000 complete [11806.919056] LustreError: 375416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974941 with bad export cookie 7452275524391167826 [11806.919574] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11806.945174] LustreError: 375416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11807.255549] Lustre: server umount lustre-MDT0001 complete [11819.190960] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11821.414778] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:02:34 (1790974954) [11831.319340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11831.765989] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11831.785183] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11831.832376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.897652] LustreError: 379867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11837.637928] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11840.997672] LustreError: 379866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.122331] LustreError: 379867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.875928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11849.311600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11854.909376] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11864.062483] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11869.635943] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11876.612408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11876.843215] LustreError: 381689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.051477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.734605] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11892.508648] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11900.134280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11900.259381] Lustre: lustre-OST0001: new disk, initializing [11900.263612] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11900.272971] Lustre: 382704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11900.353879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11906.227411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11906.245743] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11906.432396] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11908.675876] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [11920.535347] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11927.010259] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11927.011921] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.018411] LustreError: Skipped 2 previous similar messages [11927.021660] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11927.039780] Lustre: Skipped 2 previous similar messages [11927.060745] Lustre: Skipped 1 previous similar message [11930.423933] Lustre: server umount lustre-OST0001 complete [11932.150224] LustreError: 381690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.186210] LustreError: 381690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11949.025445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11949.034505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.049134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.054853] Lustre: Skipped 1 previous similar message [11951.075337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11952.623623] LustreError: 381688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.642142] LustreError: 381688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11953.941616] Lustre: server umount lustre-OST0000 complete [11957.733607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.750457] Lustre: Skipped 2 previous similar messages [11957.757443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.757455] Lustre: Skipped 3 previous similar messages [11964.032957] Lustre: server umount lustre-MDT0000 complete [11967.802578] LustreError: 381710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975102 with bad export cookie 7452275524391168708 [11967.806409] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11967.810919] LustreError: 381710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11982.304871] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11982.645954] Lustre: server umount lustre-MDT0001 complete [11995.037103] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [11997.545852] Key type lgssc unregistered [11997.907536] LNet: 384691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11997.915657] LNetError: 384691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11997.938765] LNet: Removed LNI 192.168.204.133@tcp [11998.987504] Key type .llcrypt unregistered [11998.989568] Key type ._llcrypt unregistered [12054.076996] Key type ._llcrypt registered [12054.081666] Key type .llcrypt registered [12054.194852] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [12055.231695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12055.450234] alg: No test for adler32 (adler32-zlib) [12056.633194] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [12056.936435] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [12058.624246] Key type lgssc registered [12060.274972] Lustre: Echo OBD driver; http://www.lustre.org/ [12068.288063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12068.310921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12069.738430] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12069.777530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12070.057390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12074.782792] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12086.036512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12086.209087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12086.268592] Lustre: 387489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12086.545218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.035212] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12099.833261] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12106.566610] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12112.357502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12112.360496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12112.363088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12112.376762] Lustre: Skipped 1 previous similar message [12117.480459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12117.493345] Lustre: Skipped 2 previous similar messages [12122.593746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12122.604532] Lustre: Skipped 1 previous similar message [12123.104489] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12123.389229] Lustre: server umount lustre-MDT0000 complete [12127.606584] LustreError: 387484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975262 with bad export cookie 11385640819509889514 [12127.619290] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12127.626731] LustreError: 387484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12128.323353] Lustre: server umount lustre-MDT0001 complete [12140.252211] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:07:53 (1790975273) [12148.894977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12149.951746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12155.373230] LustreError: 390005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.454993] LustreError: 390005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12156.711988] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12160.489300] LustreError: 390006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.605866] LustreError: 390005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.843031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12170.372582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12175.904836] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12184.340982] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12191.015175] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12200.686704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12201.122208] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12201.449593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12209.257439] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12219.524840] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12228.016779] Lustre: Permanently deactivating lustre-OST0000 [12228.038681] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12235.853258] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12238.572505] LustreError: 392991:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12238.593923] LustreError: 392991:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12238.616821] LustreError: 392991:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12242.122326] LustreError: 393040:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12242.129768] LustreError: 393040:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12246.680438] Lustre: server umount lustre-OST0000 complete [12250.126599] LustreError: 393289:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12250.142168] LustreError: 393289:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12255.205836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.209485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.218283] Lustre: Skipped 1 previous similar message [12255.242915] Lustre: Skipped 1 previous similar message [12260.322883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12260.329403] Lustre: Skipped 1 previous similar message [12262.712752] Lustre: server umount lustre-MDT0000 complete [12265.446828] LustreError: 390732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12265.486324] LustreError: 390732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12268.085684] LustreError: 389985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975403 with bad export cookie 11385640819509890088 [12268.102556] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12268.110302] LustreError: 389985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12279.198942] Lustre: server umount lustre-MDT0001 complete [12289.479788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12295.049456] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12297.753540] LustreError: 394451:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12304.579104] Lustre: 394789:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.133@tcp [12306.635504] Lustre: server umount MGS complete [12319.249200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12319.748517] LustreError: 395515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.774861] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12319.779642] Lustre: Skipped 1 previous similar message [12319.825830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12324.742863] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12329.952971] LustreError: 395515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12329.982484] LustreError: 395515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12333.950130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12334.263564] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12339.260855] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12346.853339] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12353.831243] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12355.890852] Lustre: Permanently reactivating lustre-OST0000 [12355.909310] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12355.922087] Lustre: Skipped 2 previous similar messages [12382.176912] LustreError: 397251:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12389.842641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12390.275420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12390.283484] Lustre: Skipped 1 previous similar message [12397.821977] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12409.551973] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12416.993751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12417.014166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.039342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12417.315598] Lustre: server umount lustre-OST0000 complete [12423.143460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12423.163705] Lustre: Skipped 1 previous similar message [12431.416779] Lustre: server umount lustre-MDT0000 complete [12433.380769] LustreError: 396243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.399453] LustreError: 396243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12435.446112] LustreError: 395495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975570 with bad export cookie 11385640819509891145 [12435.457908] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12435.461616] LustreError: 395495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12435.761718] Lustre: server umount lustre-MDT0001 complete [12446.997369] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [12450.017712] Key type lgssc unregistered [12450.261968] LNet: 399831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12450.268231] LNetError: 399831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12450.291902] LNet: Removed LNI 192.168.204.133@tcp [12451.275369] Key type .llcrypt unregistered [12451.277875] Key type ._llcrypt unregistered [12472.339516] Key type ._llcrypt registered [12472.343828] Key type .llcrypt registered [12472.485491] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [12490.263097] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [12491.901685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12491.935646] alg: No test for adler32 (adler32-zlib) [12493.047790] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [12493.395940] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [12495.112368] Key type lgssc registered [12496.199584] Lustre: Echo OBD driver; http://www.lustre.org/ [12536.487374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12536.536038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12537.939657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12537.981499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12538.075451] Lustre: lustre-MDT0000: new disk, initializing [12538.190785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12538.218031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12543.688777] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12556.483150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12556.628754] Lustre: 402988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12556.673096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12556.676995] Lustre: Skipped 1 previous similar message [12556.750397] Lustre: lustre-MDT0001: new disk, initializing [12556.876753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12556.919868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12556.933746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12562.007956] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12573.837844] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12581.982207] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12589.937131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12590.341710] Lustre: lustre-OST0000: new disk, initializing [12590.348904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12590.357384] Lustre: 404956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12590.504051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12598.519555] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12598.882643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12598.899685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12598.986045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12609.346109] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12614.210756] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12614.491362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12619.644687] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12619.828190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12624.867325] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12624.869096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.882644] LustreError: Skipped 1 previous similar message [12624.884620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12624.907021] Lustre: Skipped 1 previous similar message [12630.001356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12630.015239] Lustre: Skipped 2 previous similar messages [12635.106499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12635.115357] Lustre: Skipped 1 previous similar message [12636.128181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12636.358954] Lustre: server umount lustre-OST0000 complete [12643.812128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12643.818324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.848534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12645.346313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12647.407507] Lustre: server umount lustre-MDT0000 complete [12652.208037] LustreError: 402979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975787 with bad export cookie 1038115543456557417 [12652.240101] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12652.247130] LustreError: 402979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12652.525947] Lustre: server umount lustre-MDT0001 complete [12663.054073] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12666.221416] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:16:38 (1790975798) [12676.487212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.020289] LustreError: 407420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.069916] LustreError: 407420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12678.272254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12683.244568] LustreError: 407421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.480685] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12688.354771] LustreError: 407420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12693.502660] LustreError: 407421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.598867] LustreError: 407420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12700.263228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.966751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12708.619413] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12718.575265] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12726.467371] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12735.867603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12736.429431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12747.237760] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12757.451925] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12760.623534] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12764.251096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12771.301480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12771.313827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12771.351815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12771.356786] Lustre: Skipped 2 previous similar messages [12773.178476] Lustre: server umount lustre-OST0000 complete [12779.492247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.497885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12779.518981] Lustre: Skipped 1 previous similar message [12779.551419] Lustre: Skipped 1 previous similar message [12788.821536] Lustre: server umount lustre-MDT0000 complete [12789.730962] LustreError: 408150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.745230] LustreError: 408150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12793.511307] LustreError: 407401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975928 with bad export cookie 1038115543456558292 [12793.517637] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12793.525833] LustreError: 407401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12793.993165] Lustre: server umount lustre-MDT0001 complete [12809.683689] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [12813.083479] Key type lgssc unregistered [12813.551277] LNet: 411153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12813.583294] LNetError: 411153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12813.607321] LNet: Removed LNI 192.168.204.133@tcp [12815.242548] Key type .llcrypt unregistered [12815.245152] Key type ._llcrypt unregistered [12846.960501] Key type ._llcrypt registered [12846.962466] Key type .llcrypt registered [12847.096295] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [12847.417892] Key type .llcrypt unregistered [12847.421485] Key type ._llcrypt unregistered [12866.358803] Key type ._llcrypt registered [12866.363693] Key type .llcrypt registered [12866.484570] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [12867.666858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12868.021817] alg: No test for adler32 (adler32-zlib) [12869.339711] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [12869.639272] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [12871.315146] Key type lgssc registered [12873.229961] Lustre: Echo OBD driver; http://www.lustre.org/ [12883.379836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12883.395547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12884.886560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12890.790226] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12894.182475] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12894.200184] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12899.303975] LustreError: 413360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12901.357914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12901.838532] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12901.862170] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12901.879468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12906.980550] LustreError: 413360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12907.003226] LustreError: 413360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12909.062981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12909.368441] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12914.146813] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.167464] LustreError: 413359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12917.369349] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12924.387236] LustreError: 413354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.415361] LustreError: 413354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12924.541321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12924.840324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12929.977700] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [12944.867046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12944.894626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12944.924831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12945.379544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12947.616961] Lustre: server umount lustre-OST0000 complete [12954.698952] Lustre: server umount lustre-MDT0000 complete [12955.617637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.620169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12955.631929] Lustre: Skipped 1 previous similar message [12955.639855] LustreError: 413356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12955.671750] LustreError: 413356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12959.557670] LustreError: 413339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976094 with bad export cookie 13776113158568770503 [12959.571433] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12959.574015] LustreError: 413339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12959.910527] Lustre: server umount lustre-MDT0001 complete [12974.862417] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [12977.655398] Key type lgssc unregistered [12977.932772] LNet: 416504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12977.947858] LNetError: 416504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12977.972445] LNet: Removed LNI 192.168.204.133@tcp [12978.914906] Key type .llcrypt unregistered [12978.918500] Key type ._llcrypt unregistered [13011.762776] Key type ._llcrypt registered [13011.774173] Key type .llcrypt registered [13011.937296] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [13013.236863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13013.399442] alg: No test for adler32 (adler32-zlib) [13014.712043] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [13015.102174] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [13016.920509] Key type lgssc registered [13018.497163] Lustre: Echo OBD driver; http://www.lustre.org/ [13030.476644] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13030.508171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13046.113983] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13051.232398] LustreError: 417900:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 4: rc = -110 [13077.920257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13086.044480] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13096.578953] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13107.073211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13108.449260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13108.629777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13116.429226] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13117.936821] LustreError: 417925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13117.993956] LustreError: 417925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13123.051524] LustreError: 418935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13123.093964] LustreError: 418935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13128.164828] LustreError: 418937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13128.176730] LustreError: 418937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13129.012979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13129.690921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13135.694790] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13144.126817] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13150.263797] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13159.905731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13159.912876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13159.928435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13160.417819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13160.446060] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13160.456910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13160.463589] Lustre: Skipped 1 previous similar message [13165.546544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13165.557741] Lustre: Skipped 1 previous similar message [13170.665353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13170.673149] Lustre: Skipped 1 previous similar message [13173.216229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13173.420487] Lustre: server umount lustre-OST0000 complete [13180.904375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13180.909860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13180.928476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.934531] Lustre: Skipped 1 previous similar message [13184.677915] Lustre: server umount lustre-MDT0000 complete [13186.020172] LustreError: 420723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.054344] LustreError: 420723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13189.256237] LustreError: 420431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976324 with bad export cookie 17958535654280835904 [13189.258739] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13189.273688] LustreError: 420431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13189.893913] Lustre: server umount lustre-MDT0001 complete [13203.293762] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [13206.367582] Key type lgssc unregistered [13206.792572] LNet: 421473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13206.807866] LNetError: 421473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13206.820949] LNet: Removed LNI 192.168.204.133@tcp [13207.989886] Key type .llcrypt unregistered [13207.995853] Key type ._llcrypt unregistered [13236.313762] Key type ._llcrypt registered [13236.316527] Key type .llcrypt registered [13236.470885] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [13237.924530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13238.039159] alg: No test for adler32 (adler32-zlib) [13239.313375] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [13239.644323] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [13241.394890] Key type lgssc registered [13242.728505] Lustre: Echo OBD driver; http://www.lustre.org/ [13253.210641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13253.241860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13255.069817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13261.089507] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13265.387203] LustreError: 422895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13265.434213] LustreError: 422895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13270.501128] LustreError: 422894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13272.739496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13273.287569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13279.847295] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13288.754822] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13296.024781] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13305.095135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13305.679298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13314.133459] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13324.476620] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13325.126424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13332.858201] Lustre: server umount lustre-MDT0000 complete [13332.973890] LustreError: 424916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.028514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13335.048614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13335.071937] LustreError: 422895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.563283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13335.593107] Lustre: Skipped 1 previous similar message [13337.088473] LustreError: 422874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.33@tcp arrived at 1790976472 with bad export cookie 2921496470349340410 [13337.113592] LustreError: 422874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13339.479365] LustreError: 422875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976474 with bad export cookie 2921496470349339514 [13339.504389] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13339.512126] LustreError: 422875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13339.617875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13339.627504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13339.634742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13340.643660] LustreError: 423830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13340.646127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13340.688263] LustreError: 423830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13344.682687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13345.826986] Lustre: server umount lustre-OST0000 complete [13349.866933] LustreError: 422895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13349.898927] LustreError: 422895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13355.222860] Lustre: server umount lustre-MDT0001 complete [13369.142968] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [13373.759519] Key type lgssc unregistered [13374.143899] LNet: 426581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13374.160149] LNetError: 426581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13374.196852] LNet: Removed LNI 192.168.204.133@tcp [13375.276222] Key type .llcrypt unregistered [13375.278271] Key type ._llcrypt unregistered [13406.884818] Key type ._llcrypt registered [13406.888123] Key type .llcrypt registered [13407.126486] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [13408.769567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13408.866662] alg: No test for adler32 (adler32-zlib) [13410.679153] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [13411.359465] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [13413.232270] Key type lgssc registered [13414.879790] Lustre: Echo OBD driver; http://www.lustre.org/ [13429.229799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13429.296252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13431.148376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13438.017441] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13441.505824] LustreError: 427998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13441.526874] LustreError: 427998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13446.627771] LustreError: 427997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13448.813818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13449.407405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13456.025696] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13464.823774] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13471.919153] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13479.175862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13479.607906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13485.915389] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13493.759473] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13501.122122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13511.142325] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [13511.148188] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13511.173865] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13511.666350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13511.671390] Lustre: Skipped 1 previous similar message [13515.512290] Lustre: server umount lustre-MDT0001 complete [13516.204813] LustreError: 427994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13521.910251] LustreError: 428725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13521.955087] LustreError: 428725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13527.014557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13527.015439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13527.015845] LustreError: 427997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13527.040244] Lustre: Skipped 1 previous similar message [13527.041344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13528.532481] Lustre: server umount lustre-OST0000 complete [13533.031507] Lustre: server umount lustre-MDT0000 complete [13552.849233] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [13557.890991] Key type lgssc unregistered [13558.280802] LNet: 431703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13558.288434] LNetError: 431703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13558.322298] LNet: Removed LNI 192.168.204.133@tcp [13559.802093] Key type .llcrypt unregistered [13559.809616] Key type ._llcrypt unregistered [13597.918545] Key type ._llcrypt registered [13597.924340] Key type .llcrypt registered [13598.026108] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [13617.636688] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [13619.138554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13619.293932] alg: No test for adler32 (adler32-zlib) [13620.546154] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [13620.787510] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [13622.473731] Key type lgssc registered [13623.828970] Lustre: Echo OBD driver; http://www.lustre.org/ [13680.344221] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [13698.497065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13698.539940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13699.857851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13699.924813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13700.021072] Lustre: lustre-MDT0000: new disk, initializing [13700.136069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13700.184054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13705.346499] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13723.891717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13724.384829] Lustre: lustre-OST0000: new disk, initializing [13724.389160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13724.405691] Lustre: Skipped 1 previous similar message [13724.426131] Lustre: 436750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13724.581732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13729.939920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13729.954830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13730.000616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13733.949168] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13745.408686] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13765.862726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13766.082818] Lustre: 435732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13766.193348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13766.261387] Lustre: lustre-MDT0001: new disk, initializing [13766.386671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13766.441899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13766.450659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13771.976855] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13784.571445] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13784.850872] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13790.047757] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13790.425140] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13800.384671] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [13801.968292] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13801.969443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13801.985241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13802.000638] Lustre: Skipped 1 previous similar message [13802.022867] Lustre: Skipped 1 previous similar message [13803.335768] Lustre: server umount lustre-MDT0001 complete [13812.193706] LustreError: 438888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.228842] LustreError: 438888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13817.327167] LustreError: 435744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13817.349552] LustreError: 435744:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13822.448673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13822.451205] LustreError: 438888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.466148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13822.468670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13827.553820] LustreError: 435744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13827.554857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13827.904794] Lustre: server umount lustre-OST0000 complete [13838.826288] Lustre: server umount lustre-MDT0000 complete [13859.660905] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [13863.112570] Key type lgssc unregistered [13863.412027] LNet: 440497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13863.420655] LNetError: 440497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13863.436242] LNet: Removed LNI 192.168.204.133@tcp [13864.374183] Key type .llcrypt unregistered [13864.376234] Key type ._llcrypt unregistered [13908.377480] Key type ._llcrypt registered [13908.380231] Key type .llcrypt registered [13908.548393] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [13908.969540] Key type .llcrypt unregistered [13908.977618] Key type ._llcrypt unregistered [13929.432429] Key type ._llcrypt registered [13929.441445] Key type .llcrypt registered [13929.605312] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [13930.769813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13930.942973] alg: No test for adler32 (adler32-zlib) [13932.138706] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [13932.614680] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [13934.448255] Key type lgssc registered [13936.137593] Lustre: Echo OBD driver; http://www.lustre.org/ [13970.270386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13970.304526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13971.684802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13971.738629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13971.892603] Lustre: lustre-MDT0000: new disk, initializing [13972.019272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13972.042107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13977.523686] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [13991.969947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13992.244402] Lustre: 443871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13992.327523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13992.334684] Lustre: Skipped 1 previous similar message [13992.463624] Lustre: lustre-MDT0001: new disk, initializing [13992.674412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13992.728698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13992.740562] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13999.381723] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14011.833477] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14018.879929] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14028.798357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14029.368636] Lustre: lustre-OST0000: new disk, initializing [14029.378829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14029.397588] Lustre: 445841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14029.640143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14035.643755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14035.663856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14035.914358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14038.883131] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14052.087563] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14061.539113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14061.542554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14061.560864] Lustre: Skipped 1 previous similar message [14066.670291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14066.682689] Lustre: Skipped 4 previous similar messages [14066.835544] Lustre: server umount lustre-MDT0000 complete [14072.330771] LustreError: 443863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977207 with bad export cookie 16984062764074505615 [14072.333839] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14072.356981] LustreError: 443863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14072.956264] Lustre: server umount lustre-MDT0001 complete [14088.254601] Lustre: server umount lustre-OST0000 complete [14100.695349] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:40:34 (1790977234) [14109.418286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14110.032660] LustreError: 447869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14110.055742] LustreError: 447869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14110.154500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14114.658785] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14115.299567] LustreError: 447870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14120.427629] LustreError: 447869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14123.190275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14123.748422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14128.366776] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14137.933140] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14146.111144] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14167.797089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14168.100656] LustreError: 449695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14168.107127] Lustre: Found index 0 for lustre-OST0000, updating log [14168.462272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14176.414166] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14181.914199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14184.373731] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14192.104528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14192.122393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14192.139696] Lustre: Skipped 2 previous similar messages [14192.190979] Lustre: Skipped 1 previous similar message [14195.661667] Lustre: server umount lustre-OST0000 complete [14212.799995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14212.815391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14212.822161] Lustre: Skipped 1 previous similar message [14212.837687] Lustre: Skipped 3 previous similar messages [14218.211972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14218.222616] Lustre: Skipped 1 previous similar message [14219.245687] LustreError: 447870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14219.435293] Lustre: server umount lustre-MDT0000 complete [14225.044327] LustreError: 447850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977360 with bad export cookie 16984062764074507001 [14225.063629] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14225.072331] LustreError: 447850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14225.677512] Lustre: server umount lustre-MDT0001 complete [14233.760297] LNet: 1 local NIs in recovery (showing 1): 192.168.204.133@tcp [14237.299834] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:42:51 (1790977371) [14875.109028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14875.834383] LustreError: 454283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14875.855381] LustreError: 454283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14875.939314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14881.265214] LustreError: 454284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14881.313804] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14885.350744] LustreError: 454283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14891.494110] LustreError: 454284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14891.507184] LustreError: 454284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14893.432630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14893.812675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14899.433983] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14910.077565] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14919.774543] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14937.248195] LNet: 1 local NIs in recovery (showing 1): 192.168.204.133@tcp [14943.479944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14944.139836] Lustre: Found index 0 for lustre-OST0000, updating log [14944.811552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14950.015153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [14952.863302] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [14961.149874] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14965.736596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14965.764753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14965.766078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14969.876034] Lustre: server umount lustre-OST0000 complete [14975.980445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14975.991179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14976.020691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14976.033570] Lustre: Skipped 1 previous similar message [14978.086250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14978.095243] Lustre: Skipped 3 previous similar messages [14982.243373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14982.253362] Lustre: Skipped 7 previous similar messages [14989.280224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14989.661474] Lustre: server umount lustre-MDT0000 complete [14990.438748] LustreError: 454278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14990.465573] LustreError: 454278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14994.492913] LustreError: 454263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978129 with bad export cookie 16984062764074508093 [14994.500814] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14994.507889] LustreError: 454263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14994.939373] Lustre: server umount lustre-MDT0001 complete [14997.664369] LNet: 1 local NIs in recovery (showing 1): 192.168.204.133@tcp [15032.718952] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [15050.229381] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [15104.955222] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 17:57:17 (1790978237) [15129.551403] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [15133.244777] Key type lgssc unregistered [15133.682744] LNet: 460932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15133.700418] LNetError: 460932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15205.414837] LNet: Removed LNI 192.168.204.133@tcp [15206.525175] Key type .llcrypt unregistered [15206.529137] Key type ._llcrypt unregistered [15229.148721] Key type ._llcrypt registered [15229.151411] Key type .llcrypt registered [15229.365846] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [15231.382319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15231.515000] alg: No test for adler32 (adler32-zlib) [15232.620940] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [15232.910512] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [15234.585152] Key type lgssc registered [15236.078783] Lustre: Echo OBD driver; http://www.lustre.org/ [15242.647511] LNet: Added LNI 192.168.204.133@tcp100 [8/256/0/180] [15243.934316] LNet: Added LNI 192.168.204.133@tcp101 [8/256/0/180] [15245.255109] LNet: Added LNI 192.168.204.133@tcp102 [8/256/0/180] [15246.655423] LNet: Added LNI 192.168.204.133@tcp103 [8/256/0/180] [15248.081164] LNet: Added LNI 192.168.204.133@tcp104 [8/256/0/180] [15249.690148] LNet: Added LNI 192.168.204.133@tcp105 [8/256/0/180] [15250.832538] LNet: Added LNI 192.168.204.133@tcp106 [8/256/0/180] [15251.901533] LNet: Added LNI 192.168.204.133@tcp107 [8/256/0/180] [15253.009668] LNet: Added LNI 192.168.204.133@tcp108 [8/256/0/180] [15254.202890] LNet: Added LNI 192.168.204.133@tcp109 [8/256/0/180] [15255.465839] LNet: Added LNI 192.168.204.133@tcp110 [8/256/0/180] [15256.632343] LNet: Added LNI 192.168.204.133@tcp111 [8/256/0/180] [15257.829372] LNet: Added LNI 192.168.204.133@tcp112 [8/256/0/180] [15259.035977] LNet: Added LNI 192.168.204.133@tcp113 [8/256/0/180] [15260.396472] LNet: Added LNI 192.168.204.133@tcp114 [8/256/0/180] [15261.483181] LNet: Added LNI 192.168.204.133@tcp115 [8/256/0/180] [15262.726527] LNet: Added LNI 192.168.204.133@tcp116 [8/256/0/180] [15263.760595] LNet: Added LNI 192.168.204.133@tcp117 [8/256/0/180] [15265.039799] LNet: Added LNI 192.168.204.133@tcp118 [8/256/0/180] [15266.339406] LNet: Added LNI 192.168.204.133@tcp119 [8/256/0/180] [15267.799643] LNet: Added LNI 192.168.204.133@tcp120 [8/256/0/180] [15269.291237] LNet: Added LNI 192.168.204.133@tcp121 [8/256/0/180] [15270.654409] LNet: Added LNI 192.168.204.133@tcp122 [8/256/0/180] [15271.929149] LNet: Added LNI 192.168.204.133@tcp123 [8/256/0/180] [15273.056439] LNet: Added LNI 192.168.204.133@tcp124 [8/256/0/180] [15274.593793] LNet: Added LNI 192.168.204.133@tcp125 [8/256/0/180] [15276.106556] LNet: Added LNI 192.168.204.133@tcp126 [8/256/0/180] [15277.208765] LNet: Added LNI 192.168.204.133@tcp127 [8/256/0/180] [15278.473781] LNet: Added LNI 192.168.204.133@tcp128 [8/256/0/180] [15279.590114] LNet: Added LNI 192.168.204.133@tcp129 [8/256/0/180] [15280.899174] LNet: Added LNI 192.168.204.133@tcp130 [8/256/0/180] [15282.020462] LNet: Added LNI 192.168.204.133@tcp131 [8/256/0/180] [15283.380360] LNet: Added LNI 192.168.204.133@tcp132 [8/256/0/180] [15284.830464] LNet: Added LNI 192.168.204.133@tcp133 [8/256/0/180] [15286.283915] LNet: Added LNI 192.168.204.133@tcp134 [8/256/0/180] [15287.916406] LNet: Added LNI 192.168.204.133@tcp135 [8/256/0/180] [15297.923518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15297.964826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15299.212149] Lustre: 463958:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [15299.446777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15299.496758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15299.561625] Lustre: lustre-MDT0000: new disk, initializing [15299.677202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15299.718551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15306.353290] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [15321.446421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15321.626569] Lustre: 463974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15321.727112] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15321.730269] Lustre: Skipped 1 previous similar message [15321.871944] Lustre: lustre-MDT0001: new disk, initializing [15321.931915] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [15321.938061] Lustre: 464771:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15322.080315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15322.139852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15322.160821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15328.080615] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [15330.887927] Lustre: 465149:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [15330.897766] Lustre: 465149:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15339.761829] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15347.081119] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15354.396456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15354.839473] Lustre: lustre-OST0000: new disk, initializing [15354.843593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15354.855505] Lustre: 465960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15354.953988] Lustre: 465965:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [15354.960893] Lustre: 465965:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15355.014502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15362.379353] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [15363.642357] Lustre: 466374:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [15363.655222] Lustre: 466374:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15363.781670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15363.795865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15363.990221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15373.222633] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15384.546282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15384.549157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15384.554018] LustreError: Skipped 1 previous similar message [15384.557546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15384.582022] Lustre: Skipped 1 previous similar message [15389.343687] Lustre: server umount lustre-OST0000 complete [15395.199601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15395.214214] Lustre: Skipped 1 previous similar message [15395.223489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15395.253765] Lustre: Skipped 2 previous similar messages [15400.422094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15400.434028] Lustre: Skipped 1 previous similar message [15402.199576] Lustre: server umount lustre-MDT0000 complete [15408.505294] LustreError: 463968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978543 with bad export cookie 17392129257050490853 [15408.514201] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15408.533642] LustreError: 463968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15409.104195] Lustre: server umount lustre-MDT0001 complete [15425.497645] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [15429.823929] Key type lgssc unregistered [15430.526178] LNet: 467873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15430.544103] LNetError: 467873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15430.565543] LNet: Removed LNI 192.168.204.133@tcp [15431.584319] LNet: Removed LNI 192.168.204.133@tcp100 [15431.593997] LNetError: 467873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15431.620203] LNetError: 467873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15431.648074] LNet: Removed LNI 192.168.204.133@tcp101 [15431.652739] LNet: Removed LNI 192.168.204.133@tcp102 [15431.657469] LNet: Removed LNI 192.168.204.133@tcp103 [15431.661885] LNet: Removed LNI 192.168.204.133@tcp104 [15431.667049] LNet: Removed LNI 192.168.204.133@tcp105 [15431.676121] LNet: Removed LNI 192.168.204.133@tcp106 [15431.682456] LNet: Removed LNI 192.168.204.133@tcp107 [15431.691544] LNet: Removed LNI 192.168.204.133@tcp108 [15431.712050] LNet: Removed LNI 192.168.204.133@tcp109 [15431.725489] LNet: Removed LNI 192.168.204.133@tcp110 [15431.729043] LNet: Removed LNI 192.168.204.133@tcp111 [15431.734565] LNet: Removed LNI 192.168.204.133@tcp112 [15431.742514] LNet: Removed LNI 192.168.204.133@tcp113 [15431.745961] LNet: Removed LNI 192.168.204.133@tcp114 [15431.753717] LNet: Removed LNI 192.168.204.133@tcp115 [15431.757281] LNet: Removed LNI 192.168.204.133@tcp116 [15431.761540] LNet: Removed LNI 192.168.204.133@tcp117 [15431.763881] LNet: Removed LNI 192.168.204.133@tcp118 [15431.765910] LNet: Removed LNI 192.168.204.133@tcp119 [15431.767994] LNet: Removed LNI 192.168.204.133@tcp120 [15431.770446] LNet: Removed LNI 192.168.204.133@tcp121 [15431.773601] LNet: Removed LNI 192.168.204.133@tcp122 [15431.775990] LNet: Removed LNI 192.168.204.133@tcp123 [15431.778693] LNet: Removed LNI 192.168.204.133@tcp124 [15431.780977] LNet: Removed LNI 192.168.204.133@tcp125 [15431.783454] LNet: Removed LNI 192.168.204.133@tcp126 [15431.786125] LNet: Removed LNI 192.168.204.133@tcp127 [15431.788186] LNet: Removed LNI 192.168.204.133@tcp128 [15431.792175] LNet: Removed LNI 192.168.204.133@tcp129 [15431.794514] LNet: Removed LNI 192.168.204.133@tcp130 [15431.796711] LNet: Removed LNI 192.168.204.133@tcp131 [15431.800670] LNet: Removed LNI 192.168.204.133@tcp132 [15431.804091] LNet: Removed LNI 192.168.204.133@tcp133 [15431.806392] LNet: Removed LNI 192.168.204.133@tcp134 [15431.823835] LNet: Removed LNI 192.168.204.133@tcp135 [15432.679989] Key type .llcrypt unregistered [15432.682812] Key type ._llcrypt unregistered [15461.214881] Key type ._llcrypt registered [15461.218713] Key type .llcrypt registered [15461.361536] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [15489.345464] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [15490.857583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15490.915590] alg: No test for adler32 (adler32-zlib) [15492.355865] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [15492.900582] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [15494.729624] Key type lgssc registered [15496.852766] Lustre: Echo OBD driver; http://www.lustre.org/ [15544.443750] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:04:37 (1790978677) [15565.121371] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [15568.753548] Key type lgssc unregistered [15569.063624] LNet: 471960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15569.071948] LNetError: 471960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15569.084966] LNet: Removed LNI 192.168.204.133@tcp [15570.093441] Key type .llcrypt unregistered [15570.097498] Key type ._llcrypt unregistered [15588.632923] Key type ._llcrypt registered [15588.653367] Key type .llcrypt registered [15588.857996] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [15590.162080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15590.189845] alg: No test for adler32 (adler32-zlib) [15591.440742] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [15591.740542] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [15593.443565] Key type lgssc registered [15594.960576] Lustre: Echo OBD driver; http://www.lustre.org/ [15606.392673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15606.434928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15607.770018] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15607.796656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15607.911759] Lustre: lustre-MDT0000: new disk, initializing [15608.045563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15608.084486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15613.466882] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [15626.253977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15626.463460] Lustre: 473170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15626.563093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15626.572384] Lustre: Skipped 1 previous similar message [15626.719678] Lustre: lustre-MDT0001: new disk, initializing [15626.904282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15626.957695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15626.967393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15632.022354] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [15645.261976] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15653.848223] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15663.943478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15664.293561] Lustre: lustre-OST0000: new disk, initializing [15664.302124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15664.311624] Lustre: 475142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15664.503156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15671.663604] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [15672.522548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15672.557209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15672.778827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15682.682587] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15687.441538] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing lnet_if_list [15693.723561] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15693.751590] LNet: Added LNI 192.168.204.133@tcp10 [8/256/0/180] [15699.665965] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15699.716413] LNet: Added LNI 192.168.204.133@tcp11 [8/256/0/180] [15704.269075] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15704.313099] LNet: Added LNI 192.168.204.133@tcp12 [8/256/0/180] [15709.332791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15709.395703] LNet: Added LNI 192.168.204.133@tcp13 [8/256/0/180] [15713.607154] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15713.654669] LNet: Added LNI 192.168.204.133@tcp14 [8/256/0/180] [15718.205198] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15718.235417] LNet: Added LNI 192.168.204.133@tcp15 [8/256/0/180] [15723.810776] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15723.882653] LNet: Added LNI 192.168.204.133@tcp16 [8/256/0/180] [15729.865763] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15729.925801] LNet: Added LNI 192.168.204.133@tcp17 [8/256/0/180] [15735.595722] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15735.627843] LNet: Added LNI 192.168.204.133@tcp18 [8/256/0/180] [15741.414491] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15741.491727] LNet: Added LNI 192.168.204.133@tcp19 [8/256/0/180] [15747.550761] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15747.588674] LNet: Added LNI 192.168.204.133@tcp20 [8/256/0/180] [15753.740722] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15753.792653] LNet: Added LNI 192.168.204.133@tcp21 [8/256/0/180] [15760.167537] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15760.189964] LNet: Added LNI 192.168.204.133@tcp22 [8/256/0/180] [15768.487564] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15768.523848] LNet: Added LNI 192.168.204.133@tcp23 [8/256/0/180] [15777.372904] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15777.404280] LNet: Added LNI 192.168.204.133@tcp24 [8/256/0/180] [15789.686775] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15789.734699] LNet: Added LNI 192.168.204.133@tcp25 [8/256/0/180] [15800.527896] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15800.565796] LNet: Added LNI 192.168.204.133@tcp26 [8/256/0/180] [15809.188121] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15809.274185] LNet: Added LNI 192.168.204.133@tcp27 [8/256/0/180] [15816.927843] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15816.965233] LNet: Added LNI 192.168.204.133@tcp28 [8/256/0/180] [15825.490238] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15825.520671] LNet: Added LNI 192.168.204.133@tcp29 [8/256/0/180] [15834.329701] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15834.355767] LNet: Added LNI 192.168.204.133@tcp30 [8/256/0/180] [15843.386382] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15843.434960] LNet: Added LNI 192.168.204.133@tcp31 [8/256/0/180] [15853.462153] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15853.509548] LNet: Added LNI 192.168.204.133@tcp32 [8/256/0/180] [15862.407669] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15862.455373] LNet: Added LNI 192.168.204.133@tcp33 [8/256/0/180] [15870.803639] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15870.848442] LNet: Added LNI 192.168.204.133@tcp34 [8/256/0/180] [15878.347996] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15878.374661] LNet: Added LNI 192.168.204.133@tcp35 [8/256/0/180] [15885.866797] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15885.905084] LNet: Added LNI 192.168.204.133@tcp36 [8/256/0/180] [15894.501268] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15894.543584] LNet: Added LNI 192.168.204.133@tcp37 [8/256/0/180] [15903.365340] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15903.421484] LNet: Added LNI 192.168.204.133@tcp38 [8/256/0/180] [15914.387526] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15914.412863] LNet: Added LNI 192.168.204.133@tcp39 [8/256/0/180] [15923.444985] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15923.508500] LNet: Added LNI 192.168.204.133@tcp40 [8/256/0/180] [15931.742582] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15931.797474] LNet: Added LNI 192.168.204.133@tcp41 [8/256/0/180] [15940.238920] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15940.300200] LNet: Added LNI 192.168.204.133@tcp42 [8/256/0/180] [15948.974348] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15949.048647] LNet: Added LNI 192.168.204.133@tcp43 [8/256/0/180] [15958.857800] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15958.888950] LNet: Added LNI 192.168.204.133@tcp44 [8/256/0/180] [15969.171273] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15969.205366] LNet: Added LNI 192.168.204.133@tcp45 [8/256/0/180] [15979.789587] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15979.845862] LNet: Added LNI 192.168.204.133@tcp46 [8/256/0/180] [15990.523368] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15990.571648] LNet: Added LNI 192.168.204.133@tcp47 [8/256/0/180] [15999.839937] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15999.868338] LNet: Added LNI 192.168.204.133@tcp48 [8/256/0/180] [16008.723642] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16008.785770] LNet: Added LNI 192.168.204.133@tcp49 [8/256/0/180] [16018.298324] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16018.340391] LNet: Added LNI 192.168.204.133@tcp50 [8/256/0/180] [16041.442485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16041.458035] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16041.464335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16041.491368] Lustre: Skipped 2 previous similar messages [16042.952144] Lustre: server umount lustre-MDT0000 complete [16047.332876] LustreError: 473975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979182 with bad export cookie 6832855551063151004 [16047.342717] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16047.345555] LustreError: 473975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16047.715521] Lustre: server umount lustre-MDT0001 complete [16067.040145] Lustre: 472442:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979186/real 1790979186] req@0000000042c57a22 x1877977308781824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790979202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16067.062625] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16068.072276] Lustre: 472444:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979187/real 1790979187] req@000000006a130779 x1877977308781952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790979203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16068.501513] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [16072.161974] Lustre: 472443:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979191/real 1790979191] req@0000000097f34832 x1877977308782080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790979207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16078.741271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16079.394227] Lustre: 472442:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790979198/real 1790979198] req@00000000644c6a82 x1877977308782464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790979214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16079.425636] Lustre: 472442:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16092.708939] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5ed32debf2acdba5 [16092.737661] Lustre: MGC192.168.204.133@tcp: Connection restored to 0@lo (at 0@lo) [16093.027118] LustreError: 475136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16093.042125] LustreError: 475136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16093.080743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16096.164259] LustreError: 475137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16096.171183] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16097.185417] LustreError: 475136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16098.528853] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16102.374687] LustreError: 475137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16107.494876] LustreError: 475136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16107.631391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16108.055734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16112.196357] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16113.298736] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16122.388216] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16128.756845] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16153.075363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16153.096169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16153.101257] Lustre: Skipped 1 previous similar message [16153.115046] Lustre: Skipped 3 previous similar messages [16156.718419] Lustre: server umount lustre-OST0000 complete [16163.300551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16163.311844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16163.316165] Lustre: Skipped 1 previous similar message [16168.008710] Lustre: server umount lustre-MDT0000 complete [16168.445256] LustreError: 499393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16168.476111] LustreError: 499393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16172.048203] LustreError: 473164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979307 with bad export cookie 6832855551063153573 [16172.059614] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16172.062221] LustreError: 473164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16172.700127] Lustre: server umount lustre-MDT0001 complete [16185.178381] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [16188.256457] Key type lgssc unregistered [16188.798165] LNet: 502195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16188.806508] LNetError: 502195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16188.812817] LNet: Removed LNI 192.168.204.133@tcp [16188.814892] LNet: Removed LNI 192.168.204.133@tcp10 [16188.822927] LNet: Removed LNI 192.168.204.133@tcp11 [16188.834693] LNet: Removed LNI 192.168.204.133@tcp12 [16188.837881] LNet: Removed LNI 192.168.204.133@tcp13 [16188.849373] LNet: Removed LNI 192.168.204.133@tcp14 [16188.852607] LNet: Removed LNI 192.168.204.133@tcp15 [16188.856353] LNet: Removed LNI 192.168.204.133@tcp16 [16188.860010] LNet: Removed LNI 192.168.204.133@tcp17 [16188.863161] LNet: Removed LNI 192.168.204.133@tcp18 [16188.872532] LNet: Removed LNI 192.168.204.133@tcp19 [16188.879182] LNet: Removed LNI 192.168.204.133@tcp20 [16188.881162] LNet: Removed LNI 192.168.204.133@tcp21 [16188.889309] LNet: Removed LNI 192.168.204.133@tcp22 [16188.892392] LNet: Removed LNI 192.168.204.133@tcp23 [16188.898457] LNet: Removed LNI 192.168.204.133@tcp24 [16188.901959] LNet: Removed LNI 192.168.204.133@tcp25 [16188.905065] LNet: Removed LNI 192.168.204.133@tcp26 [16188.912829] LNet: Removed LNI 192.168.204.133@tcp27 [16188.922245] LNet: Removed LNI 192.168.204.133@tcp28 [16188.930097] LNet: Removed LNI 192.168.204.133@tcp29 [16188.936496] LNet: Removed LNI 192.168.204.133@tcp30 [16188.938908] LNet: Removed LNI 192.168.204.133@tcp31 [16188.950035] LNet: Removed LNI 192.168.204.133@tcp32 [16188.959261] LNet: Removed LNI 192.168.204.133@tcp33 [16188.970926] LNet: Removed LNI 192.168.204.133@tcp34 [16188.975191] LNet: Removed LNI 192.168.204.133@tcp35 [16188.989944] LNet: Removed LNI 192.168.204.133@tcp36 [16188.993914] LNet: Removed LNI 192.168.204.133@tcp37 [16188.999354] LNet: Removed LNI 192.168.204.133@tcp38 [16189.002417] LNet: Removed LNI 192.168.204.133@tcp39 [16189.004594] LNet: Removed LNI 192.168.204.133@tcp40 [16189.015144] LNet: Removed LNI 192.168.204.133@tcp41 [16189.017876] LNet: Removed LNI 192.168.204.133@tcp42 [16189.020459] LNet: Removed LNI 192.168.204.133@tcp43 [16189.022954] LNet: Removed LNI 192.168.204.133@tcp44 [16189.025762] LNet: Removed LNI 192.168.204.133@tcp45 [16189.028366] LNet: Removed LNI 192.168.204.133@tcp46 [16189.031187] LNet: Removed LNI 192.168.204.133@tcp47 [16189.033974] LNet: Removed LNI 192.168.204.133@tcp48 [16189.036654] LNet: Removed LNI 192.168.204.133@tcp49 [16189.045913] LNet: Removed LNI 192.168.204.133@tcp50 [16189.888306] Key type .llcrypt unregistered [16189.891469] Key type ._llcrypt unregistered [16212.474498] Key type ._llcrypt registered [16212.477793] Key type .llcrypt registered [16212.624375] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [16229.070591] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [16230.302984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16230.346696] alg: No test for adler32 (adler32-zlib) [16231.418427] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [16231.707992] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [16233.360267] Key type lgssc registered [16234.446488] Lustre: Echo OBD driver; http://www.lustre.org/ [16269.137402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16269.170700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16270.433900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16270.469445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16270.568273] Lustre: lustre-MDT0000: new disk, initializing [16270.696638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16270.716450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16275.610594] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16286.588213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16286.703712] Lustre: 505407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16286.760426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16286.770452] Lustre: Skipped 1 previous similar message [16286.916916] Lustre: lustre-MDT0001: new disk, initializing [16287.008356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16287.060153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16287.071204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16292.002236] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16301.618941] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16306.975512] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16314.902777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16315.166230] Lustre: *** cfs_fail_loc=90f, val=0*** [16315.253717] Lustre: lustre-OST0000: new disk, initializing [16315.258272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16315.264421] Lustre: 507421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16315.345482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16321.394850] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16325.710065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16325.721504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16325.839178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16332.321440] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16341.476500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16341.480063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16341.486932] LustreError: Skipped 1 previous similar message [16341.492021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16341.523229] Lustre: Skipped 1 previous similar message [16346.603226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16346.606281] Lustre: Skipped 2 previous similar messages [16347.709576] Lustre: server umount lustre-OST0000 complete [16353.765062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16353.779645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16353.794392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16356.840290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16356.841090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16358.090707] Lustre: server umount lustre-MDT0000 complete [16361.908137] LustreError: 505399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979496 with bad export cookie 2276035266891418911 [16361.912660] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16361.922495] LustreError: 505399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16362.295981] Lustre: server umount lustre-MDT0001 complete [16373.683778] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [16376.840713] Key type lgssc unregistered [16377.217568] LNet: 509328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16377.235770] LNetError: 509328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16377.261071] LNet: Removed LNI 192.168.204.133@tcp [16378.220929] Key type .llcrypt unregistered [16378.223924] Key type ._llcrypt unregistered [16399.870174] Key type ._llcrypt registered [16399.873150] Key type .llcrypt registered [16399.981348] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [16401.131097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16401.247233] alg: No test for adler32 (adler32-zlib) [16402.220767] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [16402.409832] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [16404.056201] Key type lgssc registered [16405.049160] Lustre: Echo OBD driver; http://www.lustre.org/ [16413.224170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16413.272797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16414.942421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16420.072364] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16424.419139] LustreError: 510595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16424.434260] LustreError: 510595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16429.541584] LustreError: 510594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16430.439924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16430.791534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16435.937860] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16444.594592] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16450.690378] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16457.234589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16457.576924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16463.878899] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16472.147248] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16486.854270] loop0: detected capacity change from 0 to 204800000 [16487.054145] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16487.071958] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16487.105114] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16493.522548] loop0: detected capacity change from 0 to 204800000 [16493.675050] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16493.689953] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16493.698990] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16499.197821] loop0: detected capacity change from 0 to 204800000 [16505.723163] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16505.888830] Lustre: fs15246-OST0000: new disk, initializing [16505.896990] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16505.908239] Lustre: 514191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16505.974616] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16512.648706] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16522.274317] loop1: detected capacity change from 0 to 204800000 [16527.524185] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16527.632133] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16527.684273] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16527.741683] Lustre: fs15246-MDT0000: new disk, initializing [16527.888370] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16527.916393] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16527.983901] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16529.056974] Lustre: 515315:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c674719a x1877978159038976/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790979680 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16532.469409] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16538.161840] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16538.169907] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16538.209287] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16557.545035] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16557.585164] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16562.664635] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16563.072931] Lustre: server umount fs15246-MDT0000 complete [16567.787957] LustreError: 515590:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16572.682824] Lustre: server umount fs15246-OST0000 complete [16587.746782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16587.747908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16587.755584] LustreError: Skipped 1 previous similar message [16587.771670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16587.787409] Lustre: Skipped 1 previous similar message [16589.795638] Lustre: server umount lustre-OST0000 complete [16594.914267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16594.926081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16594.940530] Lustre: Skipped 1 previous similar message [16594.947608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16594.953280] Lustre: Skipped 2 previous similar messages [16598.504609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16603.618573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16603.631143] Lustre: Skipped 3 previous similar messages [16608.228767] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16608.401546] Lustre: server umount lustre-MDT0000 complete [16608.737629] LustreError: 511322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16608.748606] LustreError: 511322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16611.866613] LustreError: 512440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979746 with bad export cookie 8950606757145328918 [16611.872199] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16611.880217] LustreError: 512440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16612.309696] Lustre: server umount lustre-MDT0001 complete [16622.060457] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [16624.738713] Key type lgssc unregistered [16625.026878] LNet: 518410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16625.036412] LNetError: 518410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16625.073561] LNet: Removed LNI 192.168.204.133@tcp [16625.988876] Key type .llcrypt unregistered [16625.990874] Key type ._llcrypt unregistered [16644.341757] Key type ._llcrypt registered [16644.344379] Key type .llcrypt registered [16644.471374] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [16645.368171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16645.395697] alg: No test for adler32 (adler32-zlib) [16646.425685] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [16646.698936] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [16648.384237] Key type lgssc registered [16649.452399] Lustre: Echo OBD driver; http://www.lustre.org/ [16683.531982] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [16692.784687] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [16726.983829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16727.013623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16728.279240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16728.310975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16728.405868] Lustre: lustre-MDT0000: new disk, initializing [16728.505760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16728.518597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16732.566747] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16742.072516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16742.216634] Lustre: 522870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16742.266392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16742.271238] Lustre: Skipped 1 previous similar message [16742.340536] Lustre: lustre-MDT0001: new disk, initializing [16742.428243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16742.468335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16742.476978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16746.831913] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16755.905549] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16761.938681] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16769.199573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16769.482536] Lustre: lustre-OST0000: new disk, initializing [16769.487258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16769.494619] Lustre: 524842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16769.591411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16775.788249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16775.797746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16775.936387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16776.061934] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [16786.848529] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16791.465455] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16791.759478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16796.954579] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16797.282453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16801.776586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16801.781557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16801.793347] LustreError: Skipped 1 previous similar message [16801.801804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16801.821054] Lustre: Skipped 1 previous similar message [16805.850193] Lustre: server umount lustre-OST0000 complete [16812.003908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16812.012438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16812.026604] Lustre: Skipped 2 previous similar messages [16812.052212] Lustre: Skipped 1 previous similar message [16815.012078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16815.017492] Lustre: Skipped 1 previous similar message [16815.439358] Lustre: server umount lustre-MDT0000 complete [16819.080641] LustreError: 522864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790979954 with bad export cookie 17688685382329293308 [16819.086015] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16819.093563] LustreError: 522864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16819.468355] Lustre: server umount lustre-MDT0001 complete [16843.325599] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [16853.712396] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [16887.080752] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16534 sec ========= 18:27:00 (1790980020) [16888.660265] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:27:02 (1790980022) === [16892.325048] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:27:06 (1790980026) === [16937.936576] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [16941.587521] Key type lgssc unregistered [16941.953307] LNet: 531484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16941.968084] LNetError: 531484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16941.999721] LNet: Removed LNI 192.168.204.133@tcp [16943.366277] Key type .llcrypt unregistered [16943.372492] Key type ._llcrypt unregistered