[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466587929 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001023] APIC: Switch to symmetric I/O mode setup [ 0.003256] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007685] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010099] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.021395] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026352] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.031427] *** VALIDATE proc *** [ 0.032177] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034209] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036239] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042196] debug: unmapping init [mem 0xffffffff8ba59000-0xffffffff8ba60fff] [ 0.044162] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050019] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.054236] rcu: Hierarchical SRCU implementation. [ 0.056458] smp: Bringing up secondary CPUs ... [ 0.057584] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.061419] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.211446] node 0 deferred pages initialised in 144ms [ 0.213293] devtmpfs: initialized [ 0.214471] x86/mm: Memory block size: 128MB [ 0.217261] gcov: version magic: 0x41383552 [ 0.219397] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.220144] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.221348] pinctrl core: initialized pinctrl subsystem [ 0.222227] [ 0.222800] ************************************************************* [ 0.223016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.224014] ** ** [ 0.225014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.226010] ** ** [ 0.227012] ** This means that this kernel is built to expose internal ** [ 0.228011] ** IOMMU data structures, which may compromise security on ** [ 0.229018] ** your system. ** [ 0.230013] ** ** [ 0.231010] ** If you see this message and you are not debugging the ** [ 0.232013] ** kernel, report this immediately to your vendor! ** [ 0.233017] ** ** [ 0.234010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.235008] ************************************************************* [ 0.236654] NET: Registered protocol family 16 [ 0.237496] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.238081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.239053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.240763] cpuidle: using governor menu [ 0.243324] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.244557] PCI: Using configuration type 1 for base access [ 0.245141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.253093] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.254048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.256038] cryptd: max_cpu_qlen set to 1000 [ 0.258224] ACPI: Added _OSI(Module Device) [ 0.259016] ACPI: Added _OSI(Processor Device) [ 0.260027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.261019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.265162] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.269176] ACPI: Interpreter enabled [ 0.270067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.271017] ACPI: Using IOAPIC for interrupt routing [ 0.272140] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.273456] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.284142] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.285125] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287180] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289561] acpiphp: Slot [2] registered [ 0.290152] acpiphp: Slot [5] registered [ 0.291130] acpiphp: Slot [6] registered [ 0.292111] acpiphp: Slot [7] registered [ 0.293120] acpiphp: Slot [8] registered [ 0.294149] acpiphp: Slot [9] registered [ 0.295130] acpiphp: Slot [10] registered [ 0.296167] acpiphp: Slot [3] registered [ 0.297112] acpiphp: Slot [4] registered [ 0.298115] acpiphp: Slot [11] registered [ 0.299120] acpiphp: Slot [12] registered [ 0.300153] acpiphp: Slot [13] registered [ 0.301111] acpiphp: Slot [14] registered [ 0.302118] acpiphp: Slot [15] registered [ 0.303190] acpiphp: Slot [16] registered [ 0.304105] acpiphp: Slot [17] registered [ 0.305182] acpiphp: Slot [18] registered [ 0.306143] acpiphp: Slot [19] registered [ 0.307205] acpiphp: Slot [20] registered [ 0.308120] acpiphp: Slot [21] registered [ 0.309117] acpiphp: Slot [22] registered [ 0.310154] acpiphp: Slot [23] registered [ 0.311131] acpiphp: Slot [24] registered [ 0.312180] acpiphp: Slot [25] registered [ 0.313130] acpiphp: Slot [26] registered [ 0.314186] acpiphp: Slot [27] registered [ 0.315127] acpiphp: Slot [28] registered [ 0.316105] acpiphp: Slot [29] registered [ 0.317113] acpiphp: Slot [30] registered [ 0.318109] acpiphp: Slot [31] registered [ 0.319085] PCI host bridge to bus 0000:00 [ 0.320025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.325035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328098] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330455] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339053] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.340028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343724] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.344520] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.346907] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.360021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.365031] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.371023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.378039] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383986] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.395022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.401022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439450] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.452031] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.462021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.484017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.497591] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.512029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.522022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566408] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.574020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.580020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.601062] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.612495] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.625022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.636026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.654024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.668609] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.670346] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.672354] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.674353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.677184] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.681042] iommu: Default domain type: Passthrough [ 0.682517] SCSI subsystem initialized [ 0.683101] ACPI: bus type USB registered [ 0.685138] usbcore: registered new interface driver usbfs [ 0.687114] usbcore: registered new interface driver hub [ 0.688110] usbcore: registered new device driver usb [ 0.689163] pps_core: LinuxPPS API ver. 1 registered [ 0.690007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.693069] PTP clock support registered [ 0.695054] EDAC MC: Ver: 3.0.0 [ 0.697162] PCI: Using ACPI for IRQ routing [ 0.699820] NetLabel: Initializing [ 0.701011] NetLabel: domain hash size = 128 [ 0.702011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704118] NetLabel: unlabeled traffic allowed by default [ 0.707152] vgaarb: loaded [ 0.709371] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715256] clocksource: Switched to clocksource kvm-clock [ 0.838137] VFS: Disk quotas dquot_6.6.0 [ 0.839300] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.841090] *** VALIDATE ramfs *** [ 0.841947] *** VALIDATE hugetlbfs *** [ 0.843118] pnp: PnP ACPI init [ 0.845124] pnp: PnP ACPI: found 6 devices [ 0.861064] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.863469] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.865047] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.866493] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.868210] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.869903] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.871930] NET: Registered protocol family 2 [ 0.873797] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.877835] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.880403] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.885295] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.888243] TCP: Hash tables configured (established 65536 bind 65536) [ 0.891682] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.894161] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897167] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899718] NET: Registered protocol family 1 [ 0.902224] RPC: Registered named UNIX socket transport module. [ 0.903547] RPC: Registered udp transport module. [ 0.905688] RPC: Registered tcp transport module. [ 0.906876] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.908632] NET: Registered protocol family 44 [ 0.909756] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.912235] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.914547] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.917983] PCI: CLS 0 bytes, default 64 [ 0.920167] Unpacking initramfs... [ 2.326382] debug: unmapping init [mem 0xffff8fccbcc54000-0xffff8fccbffbffff] [ 2.330546] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.332830] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.335526] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.854356] Initialise system trusted keyrings [ 2.855707] Key type blacklist registered [ 2.857599] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.867258] zbud: loaded [ 2.870737] *** VALIDATE nfs *** [ 2.871972] *** VALIDATE nfs4 *** [ 2.873373] pstore: using deflate compression [ 2.876885] Platform Keyring initialized [ 2.970905] NET: Registered protocol family 38 [ 2.973335] Key type asymmetric registered [ 2.974905] Asymmetric key parser 'x509' registered [ 2.977198] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.980320] io scheduler mq-deadline registered [ 2.981503] io scheduler kyber registered [ 2.982852] io scheduler bfq registered [ 2.984912] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.987825] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.991384] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.994628] ACPI: Power Button [PWRF] [ 2.999928] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.007741] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.020098] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.026512] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.042262] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.070179] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.098785] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.103771] Non-volatile memory driver v1.3 [ 3.105312] Linux agpgart interface v0.103 [ 3.139415] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.142398] vda: detected capacity change from 0 to 74702848 [ 3.163152] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.166391] vdb: detected capacity change from 0 to 1073741824 [ 3.179907] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.181806] vdc: detected capacity change from 0 to 2621440000 [ 3.193641] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.196332] vdd: detected capacity change from 0 to 2621440000 [ 3.209408] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.212337] vde: detected capacity change from 0 to 4294967296 [ 3.225419] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.228375] vdf: detected capacity change from 0 to 4294967296 [ 3.234716] libphy: Fixed MDIO Bus: probed [ 3.243921] usbcore: registered new interface driver usbserial_generic [ 3.246165] usbserial: USB Serial support registered for generic [ 3.248817] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.253384] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.255286] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.258176] mousedev: PS/2 mouse device common for all mice [ 3.260152] rtc_cmos 00:05: RTC can wake from S4 [ 3.262098] rtc_cmos 00:05: registered as rtc0 [ 3.263252] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.265723] intel_pstate: CPU model not supported [ 3.265876] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.271327] hid: raw HID events driver (C) Jiri Kosina [ 3.272316] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.273981] usbcore: registered new interface driver usbhid [ 3.277033] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.278177] usbhid: USB HID core driver [ 3.278386] drop_monitor: Initializing network drop monitor service [ 3.285190] Initializing XFRM netlink socket [ 3.287449] NET: Registered protocol family 10 [ 3.290104] Segment Routing with IPv6 [ 3.291129] NET: Registered protocol family 17 [ 3.292445] mpls_gso: MPLS GSO support [ 3.298050] RAS: Correctable Errors collector initialized. [ 3.299522] AVX version of gcm_enc/dec engaged. [ 3.300684] AES CTR mode by8 optimization enabled [ 3.366057] sched_clock: Marking stable (3366026276, 0)->(4399172568, -1033146292) [ 3.368582] registered taskstats version 1 [ 3.369898] Loading compiled-in X.509 certificates [ 3.371322] zswap: loaded using pool lzo/zbud [ 3.388212] Key type big_key registered [ 3.397369] Key type encrypted registered [ 3.398728] ima: No TPM chip found, activating TPM-bypass! [ 3.400164] ima: Allocated hash algorithm: sha1 [ 3.401406] ima: No architecture policies found [ 3.402955] evm: Initialising EVM extended attributes: [ 3.404745] evm: security.selinux [ 3.405440] evm: security.ima [ 3.406225] evm: security.capability [ 3.407282] evm: HMAC attrs: 0x1 [ 3.408852] rtc_cmos 00:05: setting system clock to 2026-08-15 21:15:43 UTC (1786828543) [ 3.414596] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.416443] debug: unmapping init [mem 0xffffffff8b782000-0xffffffff8ba58fff] [ 3.424077] Write protecting the kernel read-only data: 28672k [ 3.426197] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 3.428594] debug: unmapping init [mem 0xffffffff8a714000-0xffffffff8a7fffff] [ 3.460890] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.470186] systemd[1]: Detected virtualization kvm. [ 3.472076] systemd[1]: Detected architecture x86-64. [ 3.474097] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.495710] systemd[1]: No hostname configured. [ 3.497708] systemd[1]: Set hostname to . [ 3.499962] random: systemd: uninitialized urandom read (16 bytes read) [ 3.502900] systemd[1]: Initializing machine ID from random generator. [ 3.639780] random: systemd: uninitialized urandom read (16 bytes read) [ 3.642463] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.645845] random: systemd: uninitialized urandom read (16 bytes read) [ 3.648692] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.652924] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.299434] device-mapper: uevent: version 1.0.3 [ 4.301653] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.910051] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.109225] virtio_net virtio0 ens2: renamed from eth0 [ 5.202106] scsi host0: ata_piix [ 5.214695] scsi host1: ata_piix [ 5.216408] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.219047] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.563330] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.913220] random: crng init done [ 9.914614] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.351830] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.496686] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.772440] SELinux: Disabled at runtime. [ 11.837454] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.847299] systemd[1]: Detected virtualization kvm. [ 11.849131] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.338450] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.340803] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.345036] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.348994] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.352282] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.359963] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.364271] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.514983] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.777777] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.057981] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.116690] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.205914] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.218796] EDAC sbridge: Ver: 1.1.2 [ 14.712620] Key type dns_resolver registered [ 15.017217] NFS: Registering the id_resolver key type [ 15.019250] Key type id_resolver registered [ 15.020837] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 41.253063] libcfs: loading out-of-tree module taints kernel. [ 41.273519] Key type ._llcrypt registered [ 41.275281] Key type .llcrypt registered [ 41.325921] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 50.990342] hrtimer: interrupt took 6331058 ns [ 52.694266] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 54.656400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.665337] alg: No test for adler32 (adler32-zlib) [ 56.245845] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 57.118294] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 58.903774] Key type lgssc registered [ 60.587666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.532906] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.454916] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 129.349799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.372980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.701871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 130.772837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 130.841356] Lustre: lustre-MDT0000: new disk, initializing [ 130.942838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.965128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 135.025383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.050032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.153967] Lustre: 6486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 147.183284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 147.186139] Lustre: Skipped 1 previous similar message [ 147.299524] Lustre: lustre-MDT0001: new disk, initializing [ 147.376576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 147.407181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 147.412675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 151.020351] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 155.071627] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 163.536145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.758747] Lustre: lustre-OST0000: new disk, initializing [ 163.762216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 163.768521] Lustre: 8424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 163.833768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.638128] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.536067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 170.551350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 170.691510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 183.259835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.412697] Lustre: lustre-OST0001: new disk, initializing [ 183.417878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 183.424634] Lustre: 9494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.491786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.655798] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.086693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.103389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 193.168864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 203.045362] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.412270] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 217.482072] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 222.723893] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 228.314869] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 231.607755] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 234.148168] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 235.899873] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 15 17:19:35 EDT 2026 [ 253.473365] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 254.892235] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 275.424961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 275.429738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.448229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.454812] Lustre: Skipped 3 previous similar messages [ 277.489790] Lustre: server umount lustre-MDT0000 complete [ 281.616660] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828821 with bad export cookie 3086292736661112869 [ 281.626756] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.664086] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.664230] LustreError: 7796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.678850] Lustre: Skipped 3 previous similar messages [ 285.680976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.712113] LustreError: 7796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 286.706493] LustreError: 6494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.708796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 286.724080] LustreError: 6494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 286.745252] Lustre: Skipped 2 previous similar messages [ 288.110490] Lustre: server umount lustre-MDT0001 complete [ 291.977957] Lustre: server umount lustre-OST0000 complete [ 296.117530] Lustre: server umount lustre-OST0001 complete [ 302.731763] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 314.431653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 349.097244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.307825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 349.327957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 349.410542] Lustre: lustre-MDT0000: new disk, initializing [ 349.483022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.511439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 353.945709] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 363.705596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.789723] Lustre: 15048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 363.841102] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 363.847520] Lustre: Skipped 1 previous similar message [ 363.898555] Lustre: lustre-MDT0001: new disk, initializing [ 363.970992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 364.004386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 364.015859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 368.264296] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 378.513945] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 385.010405] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 392.842914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 393.251147] Lustre: lustre-OST0000: new disk, initializing [ 393.262268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 393.269659] Lustre: 17018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 393.426502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.671427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 394.687183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 394.783664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 400.472072] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 411.533421] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 417.541113] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 417.757390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 422.276305] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 422.475423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 425.444658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 425.449271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.457139] LustreError: Skipped 1 previous similar message [ 425.461776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 425.480884] Lustre: Skipped 2 previous similar messages [ 430.577407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 430.584341] Lustre: Skipped 2 previous similar messages [ 430.966886] Lustre: server umount lustre-OST0000 complete [ 435.680763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 435.685134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.699979] Lustre: Skipped 1 previous similar message [ 440.804601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.822090] Lustre: Skipped 3 previous similar messages [ 441.752551] Lustre: server umount lustre-MDT0000 complete [ 445.913952] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786828986 with bad export cookie 3086292736661114402 [ 445.916369] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.925254] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 446.404496] Lustre: server umount lustre-MDT0001 complete [ 450.710816] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:23:10 (1786828990) [ 453.413830] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 455.670363] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:23:15 (1786828995) [ 463.334524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.059242] LustreError: 19430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.124315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.868243] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 469.475961] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.592636] LustreError: 19430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.787276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.414733] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 488.828948] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 495.240513] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 502.583658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.698688] LustreError: 21253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.468871] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 515.140789] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.748685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.758165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.783512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.306494] Lustre: server umount lustre-OST0000 complete [ 533.475120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 533.482602] LustreError: Skipped 1 previous similar message [ 533.487538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.501487] Lustre: Skipped 1 previous similar message [ 539.784335] Lustre: server umount lustre-MDT0000 complete [ 540.128956] LustreError: 19425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.156390] LustreError: 19425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 543.600938] LustreError: 19411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829083 with bad export cookie 3086292736661115284 [ 543.608019] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.624765] LustreError: 19411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 544.091756] Lustre: server umount lustre-MDT0001 complete [ 556.918412] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 559.643212] Key type lgssc unregistered [ 559.910547] LNet: 22966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.919632] LNetError: 22966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.943293] LNet: Removed LNI 192.168.202.159@tcp [ 561.091319] Key type .llcrypt unregistered [ 561.095098] Key type ._llcrypt unregistered [ 584.522577] Key type ._llcrypt registered [ 584.524679] Key type .llcrypt registered [ 584.596442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 585.635119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 585.822110] alg: No test for adler32 (adler32-zlib) [ 586.932336] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 587.167487] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 588.888918] Key type lgssc registered [ 590.066416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 598.468516] loop: module loaded [ 598.538779] loop0: detected capacity change from 0 to 204800000 [ 598.735247] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 598.741568] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 599.051063] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.292784] loop0: detected capacity change from 0 to 204800000 [ 605.478031] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.485284] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 605.494501] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.420378] loop0: detected capacity change from 0 to 204800000 [ 615.533667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 615.550283] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.824302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 616.877274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 616.962637] Lustre: lustre-MDT0000: new disk, initializing [ 617.048863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.065948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 621.354666] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 630.007684] loop1: detected capacity change from 0 to 204800000 [ 636.061602] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 636.460380] Lustre: lustre-OST07c6: new disk, initializing [ 636.465663] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 636.470574] Lustre: Skipped 1 previous similar message [ 636.476320] Lustre: 26154:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 636.596339] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 643.128398] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 646.287804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 646.295127] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 646.329143] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 648.720652] Lustre: 25142:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 648.728537] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 648.734165] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 648.739786] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 648.746964] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 648.754550] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 665.071291] Lustre: 25142:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 665.085302] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 665.094279] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 665.105925] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 665.113832] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 665.120169] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 669.923811] Lustre: 25141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 669.950528] Lustre: 25141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 669.960090] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 669.965319] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.974043] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 669.982525] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.992483] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 670.002148] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.008920] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 670.019840] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.032104] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.040071] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.280940] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 673.294207] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 673.299858] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.312412] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.335462] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.360290] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.376975] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 673.393395] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.404350] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.418244] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.437449] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.446858] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.755422] Lustre: 25142:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 678.781272] Lustre: 25142:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 678.794090] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.813810] Lustre: 25142:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.833161] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.845331] Lustre: 25142:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.861937] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 678.873399] Lustre: 25142:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.885170] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 678.887647] Lustre: 25142:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.901284] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 678.906512] Lustre: 25142:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.223608] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 690.229323] Lustre: 25143:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 690.234052] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 690.237796] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.241197] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 690.246903] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.252273] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 690.258645] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.264321] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 690.269774] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 690.279361] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.285938] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.784907] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 738.809103] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 743.849518] Lustre: server umount lustre-OST07c6 complete [ 754.314835] Lustre: server umount lustre-MDT0000 complete [ 772.229554] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 775.564404] Key type lgssc unregistered [ 775.965280] LNet: 28715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.975716] LNetError: 28715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.010876] LNet: Removed LNI 192.168.202.159@tcp [ 777.966268] Key type .llcrypt unregistered [ 777.969201] Key type ._llcrypt unregistered [ 798.267219] Key type ._llcrypt registered [ 798.270867] Key type .llcrypt registered [ 798.384727] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 799.421875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.456276] alg: No test for adler32 (adler32-zlib) [ 800.764879] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 801.116070] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 802.849713] Key type lgssc registered [ 804.384177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 814.248995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 814.281964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.756094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.559665] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 825.324157] LustreError: 29984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 825.340686] LustreError: 29984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 830.049439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.591075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 835.269961] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 842.624508] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 849.522841] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 855.991980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 856.219350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.347829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 861.575585] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 869.438489] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 891.361910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.364499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 891.387480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 891.825114] Lustre: server umount lustre-OST0000 complete [ 896.489506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 896.492698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 896.503954] Lustre: Skipped 2 previous similar messages [ 896.526215] Lustre: Skipped 2 previous similar messages [ 901.610641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.621909] Lustre: Skipped 1 previous similar message [ 906.724256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 906.740213] Lustre: Skipped 1 previous similar message [ 910.303151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 910.691697] Lustre: server umount lustre-MDT0000 complete [ 911.842538] LustreError: 29979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 911.862306] LustreError: 29979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 915.355649] LustreError: 29965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829455 with bad export cookie 8949323937661943466 [ 915.369446] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 915.838051] Lustre: server umount lustre-MDT0001 complete [ 931.062199] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 934.935692] Key type lgssc unregistered [ 935.366930] LNet: 33723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.374041] LNetError: 33723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.432224] LNet: Removed LNI 192.168.202.159@tcp [ 937.643988] Key type .llcrypt unregistered [ 937.646154] Key type ._llcrypt unregistered [ 961.944149] Key type ._llcrypt registered [ 961.946350] Key type .llcrypt registered [ 962.104391] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 963.525412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.650097] alg: No test for adler32 (adler32-zlib) [ 964.897206] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 965.208976] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 966.999225] Key type lgssc registered [ 967.990536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.329249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.352544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.735084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.832705] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 986.081294] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.096034] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 989.714622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.074178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.394210] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1003.687784] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.953901] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.878746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1019.182968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.238410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1025.235265] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1033.049597] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.991320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.991646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1046.000194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1046.010038] Lustre: Skipped 1 previous similar message [ 1050.386527] Lustre: server umount lustre-OST0000 complete [ 1056.226719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.243298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.253657] Lustre: Skipped 2 previous similar messages [ 1056.270140] Lustre: Skipped 1 previous similar message [ 1060.880996] Lustre: server umount lustre-MDT0000 complete [ 1061.344808] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.361111] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1064.445776] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829604 with bad export cookie 11122767040184358564 [ 1064.469063] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.471801] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1064.804978] Lustre: server umount lustre-MDT0001 complete [ 1074.531975] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1077.255164] Key type lgssc unregistered [ 1077.537149] LNet: 38533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.542408] LNetError: 38533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.560887] LNet: Removed LNI 192.168.202.159@tcp [ 1078.210136] Key type .llcrypt unregistered [ 1078.213408] Key type ._llcrypt unregistered [ 1107.421424] Key type ._llcrypt registered [ 1107.432466] Key type .llcrypt registered [ 1107.619812] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1109.559312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1109.779768] alg: No test for adler32 (adler32-zlib) [ 1112.301405] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1112.621566] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1114.487777] Key type lgssc registered [ 1117.285216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1129.483676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1129.545097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.385331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.967285] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1141.733038] LustreError: 39953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.748436] LustreError: 39953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1144.582693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.907372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1148.926083] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1156.868356] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.303824] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1170.066811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1170.401140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.306958] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1176.557405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1183.703680] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1194.978773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.986539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.988291] Lustre: Skipped 2 previous similar messages [ 1195.002652] Lustre: Skipped 2 previous similar messages [ 1196.801649] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1199.840910] Lustre: server umount lustre-MDT0000 complete [ 1200.101312] LustreError: 42149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.114431] LustreError: 42149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1201.910451] LustreError: 42445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.146096] LustreError: 39933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829743 with bad export cookie 5232944530321667017 [ 1203.163830] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.165720] LustreError: 39933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1205.216954] LustreError: 39947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.217871] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.255226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1208.225333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1208.228622] Lustre: Skipped 1 previous similar message [ 1209.802530] Lustre: server umount lustre-MDT0001 complete [ 1231.813303] Lustre: server umount lustre-OST0000 complete [ 1249.171494] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1252.625083] Key type lgssc unregistered [ 1253.036857] LNet: 43815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.052410] LNetError: 43815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.066333] LNet: Removed LNI 192.168.202.159@tcp [ 1254.191905] Key type .llcrypt unregistered [ 1254.195104] Key type ._llcrypt unregistered [ 1279.982336] Key type ._llcrypt registered [ 1279.983610] Key type .llcrypt registered [ 1280.070353] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1281.633489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1281.643674] alg: No test for adler32 (adler32-zlib) [ 1282.606505] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1282.782605] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1284.447128] Key type lgssc registered [ 1285.571921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1294.806131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1294.844935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.505095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.136343] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1306.082355] LustreError: 45234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.093612] LustreError: 45234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1311.201888] LustreError: 45233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.998411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.338470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1316.723875] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1324.092863] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1329.921686] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1337.096363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.419466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.548526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1343.011257] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1351.391813] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1359.126160] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1359.336869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.346588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.407858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.414571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.422313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.443594] Lustre: Skipped 2 previous similar messages [ 1364.212306] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1364.482744] Lustre: server umount lustre-OST0000 complete [ 1382.881854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.886834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.892826] Lustre: Skipped 1 previous similar message [ 1382.900230] Lustre: Skipped 1 previous similar message [ 1386.979413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1386.983215] Lustre: Skipped 1 previous similar message [ 1388.566415] Lustre: server umount lustre-MDT0000 complete [ 1392.063191] LustreError: 45213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786829932 with bad export cookie 15325895312456576010 [ 1392.068638] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1392.073187] LustreError: 45213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1398.410498] Lustre: server umount lustre-MDT0001 complete [ 1411.675749] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1415.783919] Key type lgssc unregistered [ 1416.337719] LNet: 48943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.344710] LNetError: 48943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1417.388137] LNet: Removed LNI 192.168.202.159@tcp [ 1418.636415] Key type .llcrypt unregistered [ 1418.638329] Key type ._llcrypt unregistered [ 1446.441954] Key type ._llcrypt registered [ 1446.446982] Key type .llcrypt registered [ 1446.548871] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1447.585150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.632388] alg: No test for adler32 (adler32-zlib) [ 1448.662817] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1448.874307] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1450.551184] Key type lgssc registered [ 1452.273456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.177100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.270807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.251700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.770469] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1473.522076] LustreError: 50361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1473.556215] LustreError: 50361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1478.592073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.640133] LustreError: 50362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.210355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1484.593114] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1494.281490] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1501.473104] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1508.585772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1508.780474] LustreError: 52180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.063579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1514.486509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1515.433909] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1523.710806] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1529.483279] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1533.081899] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1545.555930] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1517h41m24s [ 1547.134556] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1550.312476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.320188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.330686] Lustre: Skipped 1 previous similar message [ 1550.349226] Lustre: Skipped 2 previous similar messages [ 1551.110130] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1552.176789] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1552.179964] Lustre: Skipped 3 previous similar messages [ 1555.399886] Lustre: server umount lustre-MDT0000 complete [ 1557.237839] LustreError: 50358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.140699] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1563.378395] LustreError: 50357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.395481] LustreError: 50357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1565.809704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.916819] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.133306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.183855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1567.166049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1571.304270] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1575.148601] Lustre: DEBUG MARKER: Wait for df (36694) ... [ 1576.939842] Lustre: DEBUG MARKER: done [ 1581.536728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1581.547636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.562125] Lustre: Skipped 1 previous similar message [ 1581.567599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1582.576025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.695287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.701349] Lustre: Skipped 3 previous similar messages [ 1595.360942] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1595.633748] Lustre: server umount lustre-OST0000 complete [ 1603.040856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1603.041038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1603.083073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1603.085721] Lustre: Skipped 2 previous similar messages [ 1606.514611] Lustre: server umount lustre-MDT0000 complete [ 1610.228307] LustreError: 50361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1610.254254] LustreError: 50361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1610.854989] LustreError: 52986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830150 with bad export cookie 5074487966757211744 [ 1610.862635] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1610.866368] LustreError: 52986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1611.375592] Lustre: server umount lustre-MDT0001 complete [ 1625.349660] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1628.270858] Key type lgssc unregistered [ 1628.521728] LNet: 55293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.536039] LNetError: 55293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.557046] LNet: Removed LNI 192.168.202.159@tcp [ 1629.369664] Key type .llcrypt unregistered [ 1629.371107] Key type ._llcrypt unregistered [ 1674.776791] Key type ._llcrypt registered [ 1674.779583] Key type .llcrypt registered [ 1674.942277] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1676.792054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.862445] alg: No test for adler32 (adler32-zlib) [ 1677.888242] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1678.180404] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1679.975171] Key type lgssc registered [ 1681.514171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1689.771579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1689.792649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.104930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1691.131580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1691.422677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1696.645571] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1706.210139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.360517] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1706.402824] Lustre: 57501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1706.683138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1711.327167] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1719.462315] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1726.154496] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1733.608312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.766539] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1733.962897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1740.543732] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1743.080467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1749.623778] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.775789] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1517h44m56s [ 1758.523120] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1765.812447] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1768.710857] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1768.997808] LustreError: 57510:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1789.281845] Lustre: lustre-MDT0000: Client cc4ab842-cc5e-4e26-b276-75c359ebe025 (at 192.168.202.59@tcp) reconnecting [ 1809.031187] LustreError: 57510:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1812.444833] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1815.519862] Lustre: DEBUG MARKER: done [ 1825.249871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.255204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1825.271088] Lustre: Skipped 1 previous similar message [ 1825.275184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.312910] Lustre: Skipped 1 previous similar message [ 1828.701560] Lustre: server umount lustre-OST0000 complete [ 1834.986860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1834.996215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.015774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1836.069038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1836.080205] Lustre: Skipped 2 previous similar messages [ 1842.145292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.154134] Lustre: Skipped 1 previous similar message [ 1846.248377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.262295] Lustre: Skipped 1 previous similar message [ 1847.263181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1847.655806] Lustre: server umount lustre-MDT0000 complete [ 1851.520260] LustreError: 59364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830391 with bad export cookie 14845445025505979199 [ 1851.522124] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1851.527155] LustreError: 59364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1851.834334] Lustre: server umount lustre-MDT0001 complete [ 1868.156517] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1872.714872] Key type lgssc unregistered [ 1873.019192] LNet: 61838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.029726] LNetError: 61838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.088846] LNet: Removed LNI 192.168.202.159@tcp [ 1875.454549] Key type .llcrypt unregistered [ 1875.460595] Key type ._llcrypt unregistered [ 1934.574737] Key type ._llcrypt registered [ 1934.580651] Key type .llcrypt registered [ 1934.710495] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1936.345914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.446919] alg: No test for adler32 (adler32-zlib) [ 1937.817363] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 1938.262688] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1940.135190] Key type lgssc registered [ 1941.731771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1945.870635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1945.896044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.230166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1947.280735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1947.527932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.625696] Lustre: Failing over lustre-MDT0000 [ 1948.827735] Lustre: server umount lustre-MDT0000 complete [ 1958.581650] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:48:17 (1786830497) [ 1966.571307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.287395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.621842] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1982.941932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.128606] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1983.226796] Lustre: 64639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1983.611110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.035671] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1998.396539] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2006.440462] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2015.050266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.355598] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2015.608546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.546580] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2025.670694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2033.002684] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2040.222269] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2050.016922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2050.023175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.034173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.553126] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.565394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2051.580881] Lustre: Skipped 2 previous similar messages [ 2055.135394] Lustre: server umount lustre-MDT0000 complete [ 2059.490067] LustreError: 64632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830599 with bad export cookie 10526278002596291355 [ 2059.508680] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.511844] LustreError: 64632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2060.177986] Lustre: server umount lustre-MDT0001 complete [ 2063.953381] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2069.036586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2077.170346] Lustre: 63565:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830601/real 1786830601] req@0000000084442300 x1873627555342592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2077.199244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.214961] Lustre: Skipped 1 previous similar message [ 2079.042136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.223352] Lustre: 63564:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830604/real 1786830604] req@00000000b2e3b21d x1873627555342720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2082.271945] Lustre: 63563:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830606/real 1786830606] req@00000000bc25e28b x1873627555342848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2085.408357] LustreError: 63561:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000122b78ad x1873627555343616/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2085.753381] LustreError: 66489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.782523] LustreError: 66489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2085.883963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.025623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2086.824596] Lustre: 63563:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830610/real 1786830610] req@00000000b4e06603 x1873627555343104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2086.891096] LustreError: 66490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.909143] LustreError: 66489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2087.923071] LustreError: 66489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2090.978047] Lustre: 63564:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830615/real 1786830615] req@000000001a6410f4 x1873627555343232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.613402] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2093.029478] LustreError: 68411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.051363] LustreError: 68411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2098.157625] LustreError: 68410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.179956] LustreError: 68410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2100.289762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.650565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.133891] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2105.828454] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2106.922093] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2113.215498] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2119.837835] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2131.433045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.445099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.483126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.570959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.573600] Lustre: Skipped 5 previous similar messages [ 2142.694454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.200806] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2143.558543] Lustre: server umount lustre-MDT0000 complete [ 2147.809572] LustreError: 68411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.851706] LustreError: 68411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2149.126932] LustreError: 64631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830689 with bad export cookie 10526278002596293630 [ 2149.134862] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.136482] LustreError: 64631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2149.520639] Lustre: server umount lustre-MDT0001 complete [ 2158.870326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2169.311213] Lustre: 63562:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830693/real 1786830693] req@00000000dd4a6718 x1873627555373696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830709 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2169.337821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.360254] Lustre: Skipped 2 previous similar messages [ 2170.334643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.496683] LustreError: 63561:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004e605256 x1873627555374720/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.036796] LustreError: 66490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.138497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.205761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2178.213884] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2180.458376] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2185.695408] Lustre: 63563:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830709/real 1786830709] req@0000000014f02904 x1873627555374464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786830725 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.717525] Lustre: 63563:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2190.166715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.455652] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2195.954131] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2203.923521] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.550509] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2221.538228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2221.550509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.567211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2221.585443] Lustre: Skipped 2 previous similar messages [ 2221.594589] Lustre: Skipped 4 previous similar messages [ 2224.475587] Lustre: server umount lustre-MDT0000 complete [ 2226.665870] LustreError: 71097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.689034] LustreError: 71097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2228.510246] LustreError: 64632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830768 with bad export cookie 10526278002596295009 [ 2228.511728] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2228.524381] LustreError: 64632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2229.052955] Lustre: server umount lustre-MDT0001 complete [ 2236.316492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2240.366638] Lustre: DEBUG MARKER: files compared the same [ 2253.061113] Lustre: server umount lustre-OST0000 complete [ 2274.700809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2277.979600] Key type lgssc unregistered [ 2278.497730] LNet: 74378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.513956] LNetError: 74378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.532467] LNet: Removed LNI 192.168.202.159@tcp [ 2279.439785] Key type .llcrypt unregistered [ 2279.441294] Key type ._llcrypt unregistered [ 2308.197777] Key type ._llcrypt registered [ 2308.201104] Key type .llcrypt registered [ 2308.356646] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2309.933585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.979698] alg: No test for adler32 (adler32-zlib) [ 2310.976241] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2311.332764] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2313.055256] Key type lgssc registered [ 2314.177367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.442460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.455699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.990371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.016447] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug +malloc all [ 2333.157627] LustreError: 75794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.176302] LustreError: 75794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2338.159704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.274839] LustreError: 75795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.449214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2342.447996] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug +malloc all [ 2350.429803] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2355.795517] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2363.120282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2363.412715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.528406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2370.326413] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug +malloc all [ 2378.475912] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2389.984282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2389.988316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.013509] Lustre: Skipped 1 previous similar message [ 2390.039190] Lustre: Skipped 1 previous similar message [ 2395.104398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.119351] Lustre: Skipped 1 previous similar message [ 2395.611072] Lustre: server umount lustre-OST0000 complete [ 2400.225114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.226045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2400.227661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2400.245656] Lustre: Skipped 1 previous similar message [ 2405.343815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.349372] Lustre: Skipped 1 previous similar message [ 2406.365218] Lustre: server umount lustre-MDT0000 complete [ 2410.351598] LustreError: 75774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786830950 with bad export cookie 2858438728354408896 [ 2410.364354] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.830995] Lustre: server umount lustre-MDT0001 complete [ 2423.087790] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2426.347150] Key type lgssc unregistered [ 2426.779642] LNet: 79331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.799837] LNetError: 79331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.821334] LNet: Removed LNI 192.168.202.159@tcp [ 2427.833789] Key type .llcrypt unregistered [ 2427.836651] Key type ._llcrypt unregistered [ 2457.941555] Key type ._llcrypt registered [ 2457.944857] Key type .llcrypt registered [ 2458.026525] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2459.355872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2459.437273] alg: No test for adler32 (adler32-zlib) [ 2460.758886] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2461.218745] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2463.000073] Key type lgssc registered [ 2464.609740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.065701] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.085589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.695412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2494.815630] LustreError: 80723:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 2521.503208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.230443] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2536.354261] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.884257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.006932] Lustre: *** cfs_fail_loc=706, val=0*** [ 2544.374153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.444332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2549.335607] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2549.738808] LustreError: 80748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.764884] LustreError: 80748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2554.852655] LustreError: 80749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.903121] LustreError: 80748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2556.924453] LustreError: 80748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2558.274713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.570728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.967605] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2573.177945] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2580.440179] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2584.036163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2584.050650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.092831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2584.551538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.667255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2589.680577] Lustre: Skipped 2 previous similar messages [ 2594.788455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.803660] Lustre: Skipped 1 previous similar message [ 2597.343243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2597.620336] Lustre: server umount lustre-OST0000 complete [ 2604.513652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.527962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.543363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2610.148810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2610.151289] Lustre: Skipped 2 previous similar messages [ 2616.289378] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2616.528116] Lustre: server umount lustre-MDT0000 complete [ 2620.385691] LustreError: 81810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.430205] LustreError: 81810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2621.039302] LustreError: 80729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831161 with bad export cookie 7970540465205187609 [ 2621.044337] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.049288] LustreError: 80729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2621.616772] Lustre: server umount lustre-MDT0001 complete [ 2636.160268] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2639.600359] Key type lgssc unregistered [ 2639.900863] LNet: 84348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2639.913334] LNetError: 84348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2639.936765] LNet: Removed LNI 192.168.202.159@tcp [ 2640.992547] Key type .llcrypt unregistered [ 2640.994075] Key type ._llcrypt unregistered [ 2669.524257] Key type ._llcrypt registered [ 2669.525426] Key type .llcrypt registered [ 2669.598961] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2671.083331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2671.190593] alg: No test for adler32 (adler32-zlib) [ 2672.645232] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 2672.957443] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2674.679195] Key type lgssc registered [ 2675.787264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2684.000714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2684.016857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.547863] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2697.015570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.416957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.762664] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2702.816726] LustreError: 86452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.845206] LustreError: 86452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2706.917426] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.931202] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2711.224551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.478132] LustreError: 86452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.494727] LustreError: 86452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2711.520979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.643837] LustreError: 86456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.657024] LustreError: 86456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2718.628312] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2721.787148] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.816065] LustreError: 86455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2728.791188] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2728.803182] Lustre: MGS: Client d9a1d70f-d26b-4585-b1ff-aa5e03cf53d6 (at 0@lo) reconnecting [ 2728.821253] LustreError: 86466:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2729.076606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.163152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2733.252340] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2749.409494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2749.423091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.449459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2749.925211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.773564] Lustre: server umount lustre-OST0000 complete [ 2760.161710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2760.172064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.179872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.179880] Lustre: Skipped 1 previous similar message [ 2764.266287] Lustre: server umount lustre-MDT0000 complete [ 2765.294260] LustreError: 86450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.318422] LustreError: 86450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2768.456819] Lustre: server umount lustre-MDT0001 complete [ 2772.176177] Lustre: server umount MGS complete [ 2784.412911] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:02:04 (1786831324) [ 2814.260992] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 2826.041322] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2861.999272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.871516] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2874.629741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.843305] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2874.881986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2874.936224] Lustre: lustre-MDT0001: new disk, initializing [ 2874.990622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2875.020436] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2875.028545] Lustre: 93649:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2875.041090] LustreError: 93649:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2879.134508] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2888.881228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.164065] Lustre: lustre-OST0000: new disk, initializing [ 2889.174607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2889.187377] Lustre: 94415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2889.272513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.878096] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2896.992243] Lustre: 94770:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d9ff7497 x1873628326044672/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786831453 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2904.161302] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2910.130572] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.161441] Lustre: MGS: Client 594293b4-1794-459a-b257-d05f9a994071 (at 0@lo) reconnecting [ 2910.225788] Lustre: 95461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2910.241308] LustreError: 93635:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2910.279336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2910.392769] Lustre: lustre-MDT0000: new disk, initializing [ 2910.588257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.617580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2910.732599] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2911.652140] Lustre: 95485:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000634af271 x1873628326051328/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786831467 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2914.809683] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2916.714442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2916.724624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2916.891266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2941.921981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.927407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.935607] Lustre: Skipped 2 previous similar messages [ 2941.946649] Lustre: Skipped 2 previous similar messages [ 2945.754375] Lustre: server umount lustre-OST0000 complete [ 2952.171044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2952.176312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2952.178128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2952.190291] Lustre: Skipped 1 previous similar message [ 2952.211318] Lustre: Skipped 1 previous similar message [ 2955.990723] Lustre: server umount lustre-MDT0000 complete [ 2957.280520] LustreError: 93625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.292201] LustreError: 93625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2965.832135] Lustre: server umount lustre-MDT0001 complete [ 2969.400290] Lustre: server umount MGS complete [ 2981.615780] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:05:21 (1786831521) [ 2997.895865] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3000.272962] Key type lgssc unregistered [ 3000.531636] LNet: 97982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3000.550253] LNetError: 97982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3001.573766] LNet: Removed LNI 192.168.202.159@tcp [ 3002.570843] Key type .llcrypt unregistered [ 3002.574207] Key type ._llcrypt unregistered [ 3019.515498] Key type ._llcrypt registered [ 3019.521326] Key type .llcrypt registered [ 3019.665842] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3021.130500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3021.175109] alg: No test for adler32 (adler32-zlib) [ 3022.326607] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 3022.757429] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3024.543176] Key type lgssc registered [ 3025.865538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3036.373181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3036.413682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.142508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.008558] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3048.434945] LustreError: 99213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.462735] LustreError: 99213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3053.539507] LustreError: 99214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.403979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.727123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3061.155840] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3069.283594] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3074.715701] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3081.397109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3081.670619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.752337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3087.114299] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3096.588032] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3109.344663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3109.357362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.371668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.370112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3110.370398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.379812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.489563] LustreError: 101036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.618666] Lustre: server umount lustre-OST0000 complete [ 3120.612170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.634534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.643028] Lustre: Skipped 2 previous similar messages [ 3125.564482] Lustre: server umount lustre-MDT0000 complete [ 3125.735375] LustreError: 99213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.758463] LustreError: 99213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3129.367200] LustreError: 99195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831669 with bad export cookie 2876217753562159798 [ 3129.375439] LustreError: 99195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3129.376339] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3129.748311] Lustre: server umount lustre-MDT0001 complete [ 3141.620900] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3144.200457] Key type lgssc unregistered [ 3144.479927] LNet: 102768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.484609] LNetError: 102768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3144.497836] LNet: Removed LNI 192.168.202.159@tcp [ 3145.376717] Key type .llcrypt unregistered [ 3145.380132] Key type ._llcrypt unregistered [ 3162.632512] Key type ._llcrypt registered [ 3162.635090] Key type .llcrypt registered [ 3162.703863] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3163.658628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3163.729784] alg: No test for adler32 (adler32-zlib) [ 3164.879995] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 3165.150826] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3166.831421] Key type lgssc registered [ 3168.210447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3174.332369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3174.337830] LustreError: 103854:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3174.346081] LustreError: 103858:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3174.353367] LustreError: 103854:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3174.361993] LustreError: 103854:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3174.369594] LustreError: 103854:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3174.400935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.894812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.416263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.611856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3189.991448] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3196.947527] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing lsmod [ 3200.092911] LustreError: 105061:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3200.093890] LustreError: 105062:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3200.112650] LustreError: 105061:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3200.118063] LustreError: 105061:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3200.128785] LustreError: 105061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3200.143836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3200.553049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.590709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3205.088596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3205.101172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3205.114755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.652030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.655857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.668571] Lustre: Skipped 1 previous similar message [ 3206.676118] Lustre: Skipped 2 previous similar messages [ 3209.764473] LustreError: 103895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.775933] LustreError: 103895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3209.864525] Lustre: server umount lustre-MDT0000 complete [ 3213.454085] LustreError: 103872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831753 with bad export cookie 5456730525773780641 [ 3213.476725] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.879333] LustreError: 104325:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000018327e50 x1873628841924096/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3214.077616] Lustre: server umount lustre-MDT0001 complete [ 3228.588712] Lustre: server umount lustre-OST0000 complete [ 3237.318467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.689060] LustreError: 106125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.759855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.643616] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3242.980633] LustreError: 106126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.086767] LustreError: 106125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.207266] LustreError: 106126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.353264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.674426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3258.434937] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3267.234961] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3273.579281] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3281.353175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3281.721285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3282.760610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3289.781494] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3297.679549] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3308.523549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.530148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.537745] LustreError: Skipped 1 previous similar message [ 3308.544141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.565823] Lustre: Skipped 1 previous similar message [ 3310.488692] Lustre: server umount lustre-OST0000 complete [ 3315.169438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3315.179527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.193804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.196186] Lustre: Skipped 1 previous similar message [ 3321.165932] Lustre: server umount lustre-MDT0000 complete [ 3323.877410] LustreError: 106120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.900959] LustreError: 106120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3324.529122] LustreError: 106860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786831864 with bad export cookie 5456730525773781572 [ 3324.547360] LustreError: 106860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3324.560461] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3325.073410] Lustre: server umount lustre-MDT0001 complete [ 3338.016365] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3340.198620] Key type lgssc unregistered [ 3340.447796] LNet: 109670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3340.458664] LNetError: 109670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3340.473968] LNet: Removed LNI 192.168.202.159@tcp [ 3341.152529] Key type .llcrypt unregistered [ 3341.153921] Key type ._llcrypt unregistered [ 3367.348240] Key type ._llcrypt registered [ 3367.356393] Key type .llcrypt registered [ 3367.461245] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3368.645211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3368.776616] alg: No test for adler32 (adler32-zlib) [ 3369.799881] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 3370.046165] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3371.679458] Key type lgssc registered [ 3373.436750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3381.701337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3381.733748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3383.287275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.837446] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3393.508124] LustreError: 111088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.529811] LustreError: 111088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3396.826353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.159407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3402.067889] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3409.951089] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3415.674509] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3422.388790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.627690] LustreError: 112908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.893209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.050021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3430.373716] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3438.688874] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3444.645438] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3448.950165] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3453.919710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3453.923740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.943755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.950958] Lustre: Skipped 1 previous similar message [ 3459.041630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3459.059126] Lustre: Skipped 1 previous similar message [ 3464.174895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3464.183641] Lustre: Skipped 1 previous similar message [ 3468.255510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3468.541977] Lustre: server umount lustre-OST0000 complete [ 3474.400641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.422342] Lustre: Skipped 1 previous similar message [ 3474.448284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.450958] Lustre: Skipped 1 previous similar message [ 3479.528849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.540700] Lustre: Skipped 1 previous similar message [ 3480.521720] Lustre: server umount lustre-MDT0000 complete [ 3484.260213] LustreError: 113635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832024 with bad export cookie 10018037235550158684 [ 3484.264905] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3484.270054] LustreError: 113635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3484.903556] Lustre: server umount lustre-MDT0001 complete [ 3497.336113] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3500.259385] Key type lgssc unregistered [ 3500.546805] LNet: 114734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3500.556325] LNetError: 114734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3500.582370] LNet: Removed LNI 192.168.202.159@tcp [ 3501.585332] Key type .llcrypt unregistered [ 3501.588513] Key type ._llcrypt unregistered [ 3523.014411] Key type ._llcrypt registered [ 3523.016397] Key type .llcrypt registered [ 3523.104615] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3524.007234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3524.133857] alg: No test for adler32 (adler32-zlib) [ 3525.302778] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 3525.510024] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3527.159514] Key type lgssc registered [ 3528.104423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3535.884594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3535.913414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.510114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.756980] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3547.619719] LustreError: 115957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.653736] LustreError: 115957:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3550.564577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.949814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3555.146656] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3563.383938] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3569.293816] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3576.740421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.286271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.396442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3583.274208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3589.734812] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3600.864613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 3600.872130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.877239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.923293] Lustre: Skipped 1 previous similar message [ 3606.014512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3606.018070] Lustre: Skipped 1 previous similar message [ 3606.264430] Lustre: server umount lustre-OST0000 complete [ 3611.124085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.148276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.150503] Lustre: Skipped 1 previous similar message [ 3616.228112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.241107] Lustre: Skipped 2 previous similar messages [ 3616.933528] Lustre: server umount lustre-MDT0000 complete [ 3620.496663] LustreError: 115938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832160 with bad export cookie 13021558963486247190 [ 3620.502534] LustreError: 116683:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3620.508423] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3620.533015] LustreError: 116683:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3620.860246] Lustre: server umount lustre-MDT0001 complete [ 3631.001472] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3634.380878] Key type lgssc unregistered [ 3634.891386] LNet: 119490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3634.902920] LNetError: 119490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3634.941238] LNet: Removed LNI 192.168.202.159@tcp [ 3635.827593] Key type .llcrypt unregistered [ 3635.838571] Key type ._llcrypt unregistered [ 3666.244574] Key type ._llcrypt registered [ 3666.252235] Key type .llcrypt registered [ 3666.486442] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3667.922082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3668.005828] alg: No test for adler32 (adler32-zlib) [ 3669.183871] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 3669.446200] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3671.095960] Key type lgssc registered [ 3672.569043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3681.194665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3681.210077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.838491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.187523] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3692.001967] LustreError: 120907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.017132] LustreError: 120907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3695.525945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.782705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3700.078255] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3707.004471] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3712.655643] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3719.169868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.581435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.513677] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3727.825525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3733.389231] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3741.130979] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3744.835602] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3744.841480] Lustre: Skipped 2 previous similar messages [ 3747.929518] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3747.950915] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3749.381801] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3749.390091] Lustre: Skipped 2 previous similar messages [ 3756.613702] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3756.623117] Lustre: Skipped 1 previous similar message [ 3762.440662] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.59@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3762.449181] Lustre: Skipped 2 previous similar messages [ 3763.249198] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.59@tcp 192.168.0.[2,10]@tcp [ 3770.370879] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.59/24@tcp in log lustre-MDT0000 [ 3770.375877] Lustre: Skipped 2 previous similar messages [ 3772.257803] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.59/24@tcp [ 3772.261674] Lustre: Skipped 1 previous similar message [ 3779.064496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3779.076054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.101473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3779.108573] Lustre: Skipped 1 previous similar message [ 3779.816258] Lustre: server umount lustre-OST0000 complete [ 3783.136739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3783.143687] LustreError: Skipped 1 previous similar message [ 3783.148789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.158850] Lustre: Skipped 1 previous similar message [ 3783.162913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.162422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.169690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.198753] Lustre: Skipped 1 previous similar message [ 3789.302904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.588674] Lustre: server umount lustre-MDT0000 complete [ 3793.176172] LustreError: 123541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832333 with bad export cookie 15415003859729250526 [ 3793.182184] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3793.184575] LustreError: 123541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3793.502271] Lustre: server umount lustre-MDT0001 complete [ 3805.806392] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3808.299989] Key type lgssc unregistered [ 3808.641925] LNet: 125712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3808.651369] LNetError: 125712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3808.689620] LNet: Removed LNI 192.168.202.159@tcp [ 3809.625447] Key type .llcrypt unregistered [ 3809.627531] Key type ._llcrypt unregistered [ 3840.961906] Key type ._llcrypt registered [ 3840.964223] Key type .llcrypt registered [ 3841.134903] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3842.641556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3842.692155] alg: No test for adler32 (adler32-zlib) [ 3843.722485] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 3843.909422] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3845.543314] Key type lgssc registered [ 3846.541743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3854.415991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3854.438972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.909494] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3855.945325] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.59/24@tcp [ 3855.954173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.695443] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3865.073263] LustreError: 127276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.091709] LustreError: 127276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3868.939359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.254450] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3869.292935] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.59/24@tcp [ 3869.349772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3873.784712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3881.580248] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3888.119931] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3897.260992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3897.627669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.770909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3904.188504] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3912.587467] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3924.456853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3924.460704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.462846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3924.486660] Lustre: Skipped 1 previous similar message [ 3928.724521] Lustre: server umount lustre-OST0000 complete [ 3934.689453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.693526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3934.716635] Lustre: Skipped 1 previous similar message [ 3934.744380] Lustre: Skipped 2 previous similar messages [ 3938.690978] Lustre: server umount lustre-MDT0000 complete [ 3939.809600] LustreError: 127276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.834485] LustreError: 127276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3942.512395] LustreError: 127257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832482 with bad export cookie 6045660297939212447 [ 3942.529838] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.530969] LustreError: 127257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3942.779321] Lustre: server umount lustre-MDT0001 complete [ 3952.767989] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3955.142043] Key type lgssc unregistered [ 3955.439235] LNet: 130868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3955.460163] LNetError: 130868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3955.513521] LNet: Removed LNI 192.168.202.159@tcp [ 3956.320195] Key type .llcrypt unregistered [ 3956.323774] Key type ._llcrypt unregistered [ 3983.129988] Key type ._llcrypt registered [ 3983.133277] Key type .llcrypt registered [ 3983.241366] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3998.484759] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3999.592075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3999.723799] alg: No test for adler32 (adler32-zlib) [ 4000.794410] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 4001.094875] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4002.768104] Key type lgssc registered [ 4003.671632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4039.130224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4039.182102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.434761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4040.477043] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4040.612209] Lustre: lustre-MDT0000: new disk, initializing [ 4040.683492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.699133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4045.163219] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4056.849291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.971848] Lustre: 134268:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4057.033418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4057.036892] Lustre: Skipped 1 previous similar message [ 4057.132698] Lustre: lustre-MDT0001: new disk, initializing [ 4057.274226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.335845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4057.346790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4061.907594] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4071.636731] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4078.113893] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4085.842302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4086.234965] Lustre: lustre-OST0000: new disk, initializing [ 4086.240807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4086.247500] Lustre: 136237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4086.329845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.452418] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4094.562049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4094.582333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4094.710944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4102.950989] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4107.326834] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4107.485593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4112.547440] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4112.718431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4115.427868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.429911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.440162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.443186] LustreError: Skipped 1 previous similar message [ 4115.478105] Lustre: Skipped 1 previous similar message [ 4120.549387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4120.569989] Lustre: Skipped 2 previous similar messages [ 4121.110707] Lustre: server umount lustre-OST0000 complete [ 4129.262811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4129.272472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.295908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4130.792089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.348916] Lustre: server umount lustre-MDT0000 complete [ 4137.462933] LustreError: 134261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832677 with bad export cookie 6856339328686766619 [ 4137.474591] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4137.790178] Lustre: server umount lustre-MDT0001 complete [ 4145.811591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.118762] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.130792] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4146.180733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.353456] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4151.275029] LustreError: 138454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.395272] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.002553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.380187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4164.641643] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4173.023717] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4179.638315] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4186.987435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4187.464326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.831489] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4202.148661] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4207.848761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4208.096147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4213.477544] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4213.822808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4224.422294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4224.676599] Lustre: lustre-OST0001: new disk, initializing [ 4224.684857] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4224.692085] Lustre: 141826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4224.786377] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4231.251846] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4233.403840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4233.410735] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4233.506905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4242.131885] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4242.664837] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4248.065386] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4248.417647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4256.145937] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91a386e6f000.ost_server_uuid 50 [ 4258.446099] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91a386e6f000.ost_server_uuid in IDLE state after 0 sec [ 4269.549631] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.563595] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.563844] LustreError: Skipped 1 previous similar message [ 4269.571476] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4269.579808] Lustre: Skipped 1 previous similar message [ 4269.603148] Lustre: Skipped 3 previous similar messages [ 4274.223239] Lustre: server umount lustre-OST0001 complete [ 4274.664545] LustreError: 140276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.687041] LustreError: 140276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4277.729306] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4277.739159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.746460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.778047] LustreError: 142262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.815711] LustreError: 142262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4283.982243] Lustre: server umount lustre-OST0000 complete [ 4288.485599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4288.491090] LustreError: Skipped 1 previous similar message [ 4288.495628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.505293] Lustre: Skipped 1 previous similar message [ 4288.507273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4288.509912] Lustre: Skipped 3 previous similar messages [ 4294.684684] Lustre: server umount lustre-MDT0000 complete [ 4295.138963] LustreError: 139695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4295.166537] LustreError: 139695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4298.654778] LustreError: 138433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786832838 with bad export cookie 6856339328686767501 [ 4298.666870] LustreError: 138433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4298.668707] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4299.268457] Lustre: server umount lustre-MDT0001 complete [ 4322.128841] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4325.612449] Key type lgssc unregistered [ 4326.051620] LNet: 144833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4326.059863] LNetError: 144833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4326.094251] LNet: Removed LNI 192.168.202.159@tcp [ 4326.996842] Key type .llcrypt unregistered [ 4326.999374] Key type ._llcrypt unregistered [ 4372.793631] Key type ._llcrypt registered [ 4372.796578] Key type .llcrypt registered [ 4372.912926] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4391.914235] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4393.929852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.950725] alg: No test for adler32 (adler32-zlib) [ 4395.349753] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 4395.730229] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4397.431177] Key type lgssc registered [ 4398.717849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4438.795393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4438.848270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.282746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4440.312033] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4440.428339] Lustre: lustre-MDT0000: new disk, initializing [ 4440.497655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.513891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4445.697361] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4456.511230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.673929] Lustre: 148991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4456.733619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4456.736729] Lustre: Skipped 1 previous similar message [ 4456.809188] Lustre: lustre-MDT0001: new disk, initializing [ 4456.915606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4456.945670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4456.952601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4460.872260] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4469.764864] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4475.939039] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4483.777414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4484.023737] Lustre: lustre-OST0000: new disk, initializing [ 4484.027053] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4484.032490] Lustre: 150959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4484.122384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.979589] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4491.868795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4491.878918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4492.006595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4500.751215] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4505.390138] Lustre: Failing over lustre-OST0000 [ 4505.654032] Lustre: server umount lustre-OST0000 complete [ 4522.975077] Lustre: 147216:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833047/real 1786833047] req@00000000edd6bda2 x1873630131684352/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786833063 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4522.976602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.006471] Lustre: 147216:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4523.043138] Lustre: Skipped 1 previous similar message [ 4524.127119] Lustre: 147215:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833047/real 1786833047] req@00000000fe9f5d7e x1873630131685248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786833063 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4525.372398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4525.862544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.884932] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4529.633302] Lustre: 147216:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833052/real 1786833052] req@000000005481ac61 x1873630131686400/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786833068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4529.657587] Lustre: 147216:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4532.518350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4532.961859] Lustre: 147217:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833057/real 1786833057] req@00000000e465b13e x1873630131687552/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786833073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4532.984025] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4533.015088] Lustre: 147217:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4534.545478] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4534.548847] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4536.659505] Lustre: Failing over lustre-MDT0000 [ 4537.126242] Lustre: server umount lustre-MDT0000 complete [ 4538.343654] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4538.359026] Lustre: Skipped 2 previous similar messages [ 4543.457195] LustreError: 148999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.482860] LustreError: 148999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4544.765684] LustreError: 150156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.576906] LustreError: 148998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4548.597310] LustreError: 148998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4553.702723] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4553.720335] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4554.719144] Lustre: 147217:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833078/real 1786833078] req@00000000adb5df2c x1873630131697920/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786833094 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4554.748567] Lustre: 147217:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4554.759096] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4556.347303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.846142] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.862474] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4564.271072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.352245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4565.240213] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4568.852299] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4569.579693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4569.584787] Lustre: Skipped 1 previous similar message [ 4569.602478] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4569.658600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4579.810125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.825595] Lustre: Skipped 1 previous similar message [ 4579.828736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4579.847084] Lustre: Skipped 1 previous similar message [ 4583.483214] Lustre: server umount lustre-OST0000 complete [ 4590.055592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.063908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.070679] Lustre: Skipped 1 previous similar message [ 4590.071048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4590.087551] Lustre: Skipped 1 previous similar message [ 4595.168867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4595.176340] Lustre: Skipped 1 previous similar message [ 4600.292219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4600.301604] Lustre: Skipped 1 previous similar message [ 4601.312058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4601.507506] Lustre: server umount lustre-MDT0000 complete [ 4605.411356] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.444293] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4605.647979] LustreError: 149785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833145 with bad export cookie 14465577776931743713 [ 4605.649673] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.661257] LustreError: 149785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4606.227569] Lustre: server umount lustre-MDT0001 complete [ 4619.364483] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4622.682861] Key type lgssc unregistered [ 4623.044035] LNet: 154622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4623.047935] LNetError: 154622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4623.056910] LNet: Removed LNI 192.168.202.159@tcp [ 4623.977126] Key type .llcrypt unregistered [ 4623.982614] Key type ._llcrypt unregistered [ 4662.841532] Key type ._llcrypt registered [ 4662.843942] Key type .llcrypt registered [ 4662.949584] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4664.432659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4664.506415] alg: No test for adler32 (adler32-zlib) [ 4665.652355] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 4666.160847] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4667.959346] Key type lgssc registered [ 4670.043874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4681.839157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4681.865744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.354919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.183916] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4693.482538] LustreError: 156143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4693.507921] LustreError: 156143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4697.407073] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4698.597202] LustreError: 156144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.284373] LustreError: 156173:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4702.302435] LustreError: 156173:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4702.311332] LustreError: 156173:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4702.674251] Lustre: server umount lustre-MDT0000 complete [ 4719.064182] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 4728.685811] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4762.512106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.707876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4762.726496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4762.781561] Lustre: lustre-MDT0000: new disk, initializing [ 4762.835672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.847192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4766.531418] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4776.937039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.056646] Lustre: 159480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4777.106806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4777.117492] Lustre: Skipped 1 previous similar message [ 4777.227522] Lustre: lustre-MDT0001: new disk, initializing [ 4777.363414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4777.398552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4777.403669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4781.332911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4789.885887] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4794.640211] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4801.086461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.305213] Lustre: lustre-OST0000: new disk, initializing [ 4801.310328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4801.314338] Lustre: 161447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4801.389521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4806.919506] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4808.736419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4808.748134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4808.828179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4814.906881] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4818.107514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4819.525868] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:35:59 (1786833359) [ 5049.316702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.319813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5049.337188] Lustre: Skipped 2 previous similar messages [ 5049.352657] Lustre: Skipped 2 previous similar messages [ 5054.438723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5054.446491] Lustre: Skipped 2 previous similar messages [ 5055.642256] Lustre: server umount lustre-MDT0000 complete [ 5059.556378] LustreError: 159491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.599314] LustreError: 159491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5059.837943] LustreError: 159470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833599 with bad export cookie 6362686793457658373 [ 5059.841038] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5059.849266] LustreError: 159470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5064.679799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.689675] LustreError: 162557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5064.718649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5069.795844] LustreError: 159487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.796974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5074.399143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5074.760669] Lustre: server umount lustre-MDT0001 complete [ 5085.725153] Lustre: server umount lustre-OST0000 complete [ 5096.752300] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5107.139260] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5142.664109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.916577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5142.941120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5143.024182] Lustre: lustre-MDT0000: new disk, initializing [ 5143.093510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5143.106122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5148.686865] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5162.102609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.281795] Lustre: 166105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5162.395100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5162.400338] Lustre: Skipped 1 previous similar message [ 5162.578278] Lustre: lustre-MDT0001: new disk, initializing [ 5162.690732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5162.726631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5162.747779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5168.731198] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5180.576952] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5189.760230] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5197.739625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5197.993207] Lustre: lustre-OST0000: new disk, initializing [ 5197.996491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5198.008572] Lustre: 168073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5198.104989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.575673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5203.586015] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5203.718659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5203.939914] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5214.402167] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5219.742770] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5219.977199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5223.972043] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5224.178285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5229.536240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5229.542469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.550583] LustreError: Skipped 1 previous similar message [ 5229.564418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.572490] Lustre: Skipped 2 previous similar messages [ 5239.804579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5239.807467] Lustre: Skipped 3 previous similar messages [ 5240.287146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5240.490821] Lustre: server umount lustre-OST0000 complete [ 5244.900573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.906048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5244.925590] Lustre: Skipped 1 previous similar message [ 5258.207233] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5258.481981] Lustre: server umount lustre-MDT0000 complete [ 5260.269860] LustreError: 166117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.283640] LustreError: 166117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5261.371714] LustreError: 166098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833801 with bad export cookie 6362686793457817203 [ 5261.379558] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.380439] LustreError: 166098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5261.688494] Lustre: server umount lustre-MDT0001 complete [ 5269.441673] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:43:29 (1786833809) [ 5282.600732] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5293.057778] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5329.430977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.629668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5329.655323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5329.738673] Lustre: lustre-MDT0000: new disk, initializing [ 5329.803764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.823449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5334.702478] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5346.312069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.510644] Lustre: 172830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5346.954648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5350.988840] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5359.917612] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5365.889946] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5372.875031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.125957] Lustre: lustre-OST0000: new disk, initializing [ 5373.127282] Lustre: Skipped 1 previous similar message [ 5373.129377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5373.131034] Lustre: Skipped 2 previous similar messages [ 5373.133730] Lustre: 174807:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5379.306211] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5379.632327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5379.638115] Lustre: Skipped 1 previous similar message [ 5379.647483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5379.727163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5387.741594] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5394.918656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.922154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5394.934838] Lustre: Skipped 2 previous similar messages [ 5394.946645] Lustre: Skipped 7 previous similar messages [ 5400.042162] LustreError: 175190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.058146] LustreError: 175190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5400.140650] Lustre: server umount lustre-OST0000 complete [ 5405.153408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.169821] Lustre: Skipped 1 previous similar message [ 5411.394625] Lustre: server umount lustre-MDT0000 complete [ 5416.965724] LustreError: 172822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786833957 with bad export cookie 6362686793457818078 [ 5416.991636] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.003020] LustreError: 172822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5417.331274] Lustre: server umount lustre-MDT0001 complete [ 5427.436956] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:46:06 (1786833966) [ 5445.932834] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 5456.635481] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5491.463818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5491.680843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5491.706838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5491.791940] Lustre: lustre-MDT0000: new disk, initializing [ 5491.864014] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.870110] Lustre: Skipped 2 previous similar messages [ 5491.882834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5496.261301] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5507.040595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.261226] Lustre: 179134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5507.625880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5512.054255] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5521.891518] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5527.477608] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5534.324299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5534.621168] Lustre: 181112:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5540.968134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5541.167130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5541.907968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5553.081737] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5561.831892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5561.833844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.845562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.845572] Lustre: Skipped 4 previous similar messages [ 5561.883561] Lustre: Skipped 1 previous similar message [ 5566.264651] Lustre: server umount lustre-OST0000 complete [ 5576.359120] Lustre: server umount lustre-MDT0000 complete [ 5577.194537] LustreError: 179142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.206611] LustreError: 179142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5580.125075] LustreError: 179128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834120 with bad export cookie 6362686793457819163 [ 5580.130736] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.147036] LustreError: 179128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5580.601304] Lustre: server umount lustre-MDT0001 complete [ 5591.900492] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5594.434792] Key type lgssc unregistered [ 5594.695478] LNet: 182973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5594.699651] LNetError: 182973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5594.711845] LNet: Removed LNI 192.168.202.159@tcp [ 5595.415660] Key type .llcrypt unregistered [ 5595.418775] Key type ._llcrypt unregistered [ 5618.241315] Key type ._llcrypt registered [ 5618.243629] Key type .llcrypt registered [ 5618.315124] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5619.384435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5619.408780] alg: No test for adler32 (adler32-zlib) [ 5620.458190] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 5620.631774] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5622.271146] Key type lgssc registered [ 5623.428591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.274978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5629.311266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5630.617384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.804191] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5640.163207] LustreError: 184390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.175813] LustreError: 184390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5640.890530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.175747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5644.907924] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5651.295881] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5656.435686] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5662.345411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5662.687693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5668.583549] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5675.000241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5676.166758] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5695.458744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.476527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5695.483900] Lustre: Skipped 1 previous similar message [ 5700.430394] Lustre: server umount lustre-OST0000 complete [ 5705.696853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.702546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5705.703501] Lustre: Skipped 2 previous similar messages [ 5705.720145] Lustre: Skipped 1 previous similar message [ 5710.685633] Lustre: server umount lustre-MDT0000 complete [ 5710.824566] LustreError: 185117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.027948] LustreError: 184370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834254 with bad export cookie 14419931923788247277 [ 5714.040746] LustreError: 184370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5714.040940] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5714.372804] Lustre: server umount lustre-MDT0001 complete [ 5726.174756] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5728.704754] Key type lgssc unregistered [ 5728.987787] LNet: 187950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5728.997391] LNetError: 187950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5729.012923] LNet: Removed LNI 192.168.202.159@tcp [ 5729.960429] Key type .llcrypt unregistered [ 5729.976465] Key type ._llcrypt unregistered [ 5755.912600] Key type ._llcrypt registered [ 5755.914282] Key type .llcrypt registered [ 5756.054720] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5757.052245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5757.211431] alg: No test for adler32 (adler32-zlib) [ 5758.174423] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 5758.380364] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5760.063164] Key type lgssc registered [ 5761.040868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5769.070677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5769.095927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.618982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.471405] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5779.941821] LustreError: 189367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.969938] LustreError: 189367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5783.534648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.831039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.155441] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5796.051788] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5802.288425] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.182706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5808.491428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.700297] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5815.686436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5821.570303] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5828.372591] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 5830.124257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5830.132805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5830.173678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.138140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5831.147875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.459319] Lustre: lustre-OST0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 5833.469642] Lustre: Skipped 2 previous similar messages [ 5833.876609] Lustre: server umount lustre-OST0000 complete [ 5842.858121] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff91a385533800.ost_server_uuid 50 [ 5845.223844] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5860.832371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5860.841335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.857550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5865.409146] Lustre: server umount lustre-MDT0000 complete [ 5866.976844] LustreError: 191010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.988462] LustreError: 191010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5868.988721] LustreError: 189349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834409 with bad export cookie 5608996608475758925 [ 5868.993994] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5868.997984] LustreError: 189349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5869.279377] Lustre: server umount lustre-MDT0001 complete [ 5878.330361] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:53:38 (1786834418) [ 5884.312609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.657248] LustreError: 193282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.710341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5889.665744] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5890.021658] LustreError: 193283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.135447] LustreError: 193282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.522872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.914340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5901.281232] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5907.374238] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5912.605802] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5919.354292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5919.783202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5926.420968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5929.019123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5933.894569] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5940.399822] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5940.592256] Lustre: lustre-OST0001: new disk, initializing [ 5940.595361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5940.599902] Lustre: 196123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5940.692204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5946.252653] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5950.016779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5950.026197] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5950.093372] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5954.366026] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5958.625579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5958.633363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5958.645741] Lustre: Skipped 1 previous similar message [ 5958.658845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5958.665285] Lustre: Skipped 2 previous similar messages [ 5962.483733] Lustre: server umount lustre-OST0000 complete [ 5965.049806] LustreError: 195103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.318381] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5967.535691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5971.395205] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5971.640575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5981.157280] LustreError: 195102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.177618] LustreError: 195102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5986.272250] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5986.282089] LustreError: Skipped 1 previous similar message [ 5986.287047] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.289288] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5986.298841] Lustre: Skipped 2 previous similar messages [ 5986.305530] Lustre: Skipped 5 previous similar messages [ 5991.525723] Lustre: server umount lustre-OST0001 complete [ 5995.488268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5995.491818] LustreError: Skipped 1 previous similar message [ 5995.493911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.761585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6006.766663] Lustre: Skipped 6 previous similar messages [ 6009.311604] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6009.458883] Lustre: server umount lustre-MDT0000 complete [ 6013.022180] LustreError: 193263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834553 with bad export cookie 5608996608475759996 [ 6013.026664] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6013.033898] LustreError: 193263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6013.373327] Lustre: server umount lustre-MDT0001 complete [ 6032.127618] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:56:12 (1786834572) [ 6037.481577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6037.742740] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6037.794591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6037.942805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6041.471848] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6048.738185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.830557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6048.874329] Lustre: 199561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6052.336984] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6059.258091] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6064.536791] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6070.847142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.993201] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6077.164413] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6077.523655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6083.782252] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6090.326345] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6090.377471] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6096.047336] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6096.116023] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6103.179801] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6106.603674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.607585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6106.623202] Lustre: Skipped 2 previous similar messages [ 6106.630992] Lustre: Skipped 1 previous similar message [ 6111.946950] Lustre: server umount lustre-OST0000 complete [ 6114.034690] LustreError: 201414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.061914] LustreError: 201414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6127.074953] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.090724] LustreError: Skipped 1 previous similar message [ 6133.274420] Lustre: server umount lustre-OST0001 complete [ 6143.440569] Lustre: server umount lustre-MDT0000 complete [ 6147.736302] LustreError: 200307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834687 with bad export cookie 5608996608475761403 [ 6147.740638] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.754972] LustreError: 200307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6148.302382] Lustre: server umount lustre-MDT0001 complete [ 6173.916525] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:58:33 (1786834713) [ 6190.944123] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 6200.065537] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6231.864565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6232.059792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6232.089555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6232.144657] Lustre: lustre-MDT0000: new disk, initializing [ 6232.201559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.206236] Lustre: Skipped 3 previous similar messages [ 6232.215411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6235.907657] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6245.371125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.522915] Lustre: 207652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6245.628560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6245.634558] Lustre: Skipped 1 previous similar message [ 6245.771429] Lustre: lustre-MDT0001: new disk, initializing [ 6245.976203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6245.984044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6250.589747] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6259.522851] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6264.619128] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6270.923411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.194429] Lustre: lustre-OST0000: new disk, initializing [ 6271.204182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6271.215887] Lustre: 209624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6278.010522] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6279.767594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6279.783156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6279.962351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6286.881538] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6290.815787] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6291.001428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6294.461958] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6294.617545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6300.648321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6300.651407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.653807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.664880] Lustre: Skipped 5 previous similar messages [ 6300.673893] Lustre: Skipped 11 previous similar messages [ 6302.479981] Lustre: server umount lustre-OST0000 complete [ 6312.433352] Lustre: server umount lustre-MDT0000 complete [ 6315.242362] LustreError: 209623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786834855 with bad export cookie 5608996608475763188 [ 6315.245696] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6315.253133] LustreError: 209623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6315.554257] Lustre: server umount lustre-MDT0001 complete [ 6322.818185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.154264] LustreError: 211850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6323.173942] LustreError: 211850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6327.914831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6335.778249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.164206] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6347.587867] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.448833] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.758451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.431498] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6369.663518] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6374.437874] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6374.615358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6378.498156] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6378.649978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6381.536829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6381.547253] LustreError: Skipped 2 previous similar messages [ 6381.555538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.574614] Lustre: Skipped 2 previous similar messages [ 6386.490256] Lustre: server umount lustre-OST0000 complete [ 6391.890505] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6396.172648] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6424.594254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.842256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6424.853651] Lustre: Skipped 5 previous similar messages [ 6430.269695] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6433.064484] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.087908] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6437.664995] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6443.354793] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6448.353446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6448.364444] Lustre: Skipped 7 previous similar messages [ 6449.471490] Lustre: server umount lustre-OST0000 complete [ 6459.574653] Lustre: server umount lustre-MDT0000 complete [ 6463.164131] LustreError: 211831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835003 with bad export cookie 5608996608475764070 [ 6463.173414] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.175666] LustreError: 211831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6463.674020] Lustre: server umount lustre-MDT0001 complete [ 6472.526526] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:03:32 (1786835012) [ 6479.789785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.183473] LustreError: 217814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.203275] LustreError: 217814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6484.802532] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6494.272414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.658802] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6506.580266] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6512.742955] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6519.237610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.298219] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6534.110829] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.436805] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6538.646608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6542.998673] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6543.191631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6550.325967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.561979] Lustre: lustre-OST0001: new disk, initializing [ 6550.568376] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6550.575726] Lustre: 221183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6558.019880] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6558.354477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6558.365456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6558.486252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6567.230916] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6572.194796] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6572.509354] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6576.963810] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6577.209661] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6579.168996] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.171454] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.175656] LustreError: Skipped 3 previous similar messages [ 6579.203141] Lustre: Skipped 5 previous similar messages [ 6584.972674] Lustre: server umount lustre-OST0001 complete [ 6590.055915] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6590.186878] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6594.328169] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6594.560817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6623.141100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.330032] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6628.344817] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6628.355212] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6629.472676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6636.111832] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6637.425413] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6645.124570] Lustre: server umount lustre-OST0001 complete [ 6674.405224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6674.512812] Lustre: server umount lustre-OST0000 complete [ 6683.540878] Lustre: server umount lustre-MDT0000 complete [ 6686.336519] LustreError: 217795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835226 with bad export cookie 5608996608475765029 [ 6686.340953] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.342796] LustreError: 217795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6692.680501] Lustre: server umount lustre-MDT0001 complete [ 6712.616498] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:07:32 (1786835252) [ 6717.574910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.821570] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6717.861545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6718.002758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6718.008506] Lustre: Skipped 5 previous similar messages [ 6721.020563] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6726.508445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.574135] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6726.590885] Lustre: 226423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6729.101563] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6733.424922] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6737.121267] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6742.612280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6742.727907] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6748.123750] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6754.955772] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6762.972235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.028564] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6768.768639] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6774.379281] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6775.353495] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6779.452886] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6779.684904] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6782.765534] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6782.962203] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6787.642506] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91a384b0d800.ost_server_uuid 50 [ 6788.997094] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff91a384b0d800.ost_server_uuid in FULL state after 0 sec [ 6790.077109] Lustre: Permanently deactivating lustre-OST0001 [ 6790.100765] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6792.856840] Lustre: Permanently reactivating lustre-OST0001 [ 6796.267723] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6796.273486] Lustre: Skipped 21 previous similar messages [ 6801.998768] Lustre: server umount lustre-OST0001 complete [ 6806.512469] LustreError: 228267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.530891] LustreError: 228267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 6811.456186] Lustre: server umount lustre-OST0000 complete [ 6820.932662] Lustre: server umount lustre-MDT0000 complete [ 6824.066827] LustreError: 226414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835364 with bad export cookie 5608996608475766310 [ 6824.068375] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6824.075122] LustreError: 226414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6824.411855] Lustre: server umount lustre-MDT0001 complete [ 6844.134804] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:09:44 (1786835384) [ 6849.855601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.139568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6850.159976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6850.165721] Lustre: Skipped 5 previous similar messages [ 6853.493949] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6860.588180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.664102] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6860.687286] Lustre: 233114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6865.193505] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6872.129490] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6877.130248] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6882.852065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.986625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6882.999482] Lustre: Permanently deactivating lustre-OST0000 [ 6883.007256] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6888.140435] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6893.082163] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6893.151449] LustreError: 235600:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6894.380893] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6899.592895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6899.629935] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6901.141786] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6905.068385] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6911.375509] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6912.787836] Lustre: Permanently reactivating lustre-OST0000 [ 6912.795333] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6912.797538] Lustre: Skipped 2 previous similar messages [ 6920.250996] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6920.253594] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6920.257421] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6920.259556] Lustre: Skipped 1 previous similar message [ 6920.295829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6920.301640] Lustre: Skipped 12 previous similar messages [ 6920.306747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6920.317590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6930.342500] Lustre: server umount lustre-OST0001 complete [ 6931.943748] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6931.953402] LustreError: Skipped 5 previous similar messages [ 6939.704650] Lustre: server umount lustre-OST0000 complete [ 6949.013733] Lustre: server umount lustre-MDT0000 complete [ 6951.685124] LustreError: 233107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835491 with bad export cookie 5608996608475767969 [ 6951.692077] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6951.960405] Lustre: server umount lustre-MDT0001 complete [ 6960.416200] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6962.645851] Key type lgssc unregistered [ 6962.867084] LNet: 237997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6962.872918] LNetError: 237997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6962.889401] LNet: Removed LNI 192.168.202.159@tcp [ 6963.563544] Key type .llcrypt unregistered [ 6963.565270] Key type ._llcrypt unregistered [ 6978.726225] Key type ._llcrypt registered [ 6978.728770] Key type .llcrypt registered [ 6978.803394] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6979.324806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6979.391339] alg: No test for adler32 (adler32-zlib) [ 6980.327052] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 6980.463364] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6982.103306] Key type lgssc registered [ 6982.761429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6988.239492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6988.250944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.529510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6991.528774] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6997.704168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.814561] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6997.837296] Lustre: 239432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6997.843647] Lustre: Permanently deactivating lustre-MDT0001 [ 6997.854520] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6998.096427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7001.496133] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7003.657481] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7006.883994] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7010.958899] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7016.095142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.316865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7020.358723] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7023.462728] LustreError: 241282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.478549] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7023.476135] LustreError: 241282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7025.193897] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7028.709395] LustreError: 241859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7030.192564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7030.349482] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.476118] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7034.492126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7034.503859] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7034.862230] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7040.859269] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7044.188721] Lustre: Permanently reactivating lustre-MDT0001 [ 7044.201338] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7044.204996] Lustre: Skipped 2 previous similar messages [ 7050.506946] Lustre: lustre-MDT0001: Client b73972c7-451d-4954-a471-dd0ec6c4bff1 (at 192.168.202.59@tcp) reconnecting [ 7053.357954] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.373798] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7053.380454] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7053.386900] LustreError: 243154:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7053.391234] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7053.395647] LustreError: 243154:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7053.395774] LustreError: 243154:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -108 [ 7053.395797] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7053.395797] [ 7053.400146] LustreError: 239435:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7054.747741] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.870267] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7059.489254] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7059.638996] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7063.087911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7063.263946] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7065.308380] Lustre: Permanently deactivating lustre-MDT0001 [ 7065.323291] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7065.327172] Lustre: Skipped 2 previous similar messages [ 7074.386365] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7078.816763] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.820463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7078.840310] Lustre: Skipped 3 previous similar messages [ 7078.851949] Lustre: Skipped 3 previous similar messages [ 7084.005336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7084.008206] Lustre: Skipped 1 previous similar message [ 7084.512834] Lustre: server umount lustre-MDT0000 complete [ 7087.171979] LustreError: 239424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835627 with bad export cookie 15219552699022710882 [ 7087.184716] LustreError: 239424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7087.186619] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.377713] Lustre: server umount lustre-MDT0001 complete [ 7095.921346] Lustre: server umount lustre-OST0000 complete [ 7103.937690] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:14:04 (1786835644) [ 7111.391704] Lustre: server umount lustre-OST0001 complete [ 7116.032496] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7121.804937] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7142.190096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.379694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7142.384343] Lustre: Skipped 2 previous similar messages [ 7142.400408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7142.453851] Lustre: lustre-MDT0000: new disk, initializing [ 7142.508305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7142.528832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7145.282231] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7152.986545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.100161] Lustre: 247696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7153.143837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7153.147664] Lustre: Skipped 1 previous similar message [ 7153.211773] Lustre: lustre-MDT0001: new disk, initializing [ 7153.281491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7153.305594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7153.313241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7156.403251] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7162.162035] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7164.986653] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7169.034205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7169.248655] Lustre: lustre-OST0000: new disk, initializing [ 7169.252352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7169.261784] Lustre: 249664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7169.325879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.981648] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7176.224256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7176.231673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7176.330577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7179.210407] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7180.656358] LustreError: 247702:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7186.795423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.939653] Lustre: lustre-OST0001: new disk, initializing [ 7186.944204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7186.950092] Lustre: 250778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7190.692242] LustreError: 247702:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7190.719873] LustreError: 247701:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7190.967671] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7193.672068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7193.682456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7193.746724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7198.304127] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.404543] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7200.759110] LustreError: 247701:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7203.578241] Lustre: lustre-OST0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 7203.581395] Lustre: Skipped 1 previous similar message [ 7204.328299] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7204.329369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7208.121559] Lustre: server umount lustre-OST0001 complete [ 7208.691120] LustreError: 250201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.565958] LustreError: 249646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.574295] LustreError: 249646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7219.686871] LustreError: 250201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7219.692711] LustreError: 250201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7222.241879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7222.248457] LustreError: Skipped 1 previous similar message [ 7222.251439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7222.261095] Lustre: Skipped 1 previous similar message [ 7222.264911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7222.267293] Lustre: Skipped 2 previous similar messages [ 7228.187240] Lustre: server umount lustre-OST0000 complete [ 7230.432268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7230.435631] LustreError: Skipped 1 previous similar message [ 7230.437790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7230.445901] Lustre: Skipped 1 previous similar message [ 7230.451090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7230.453546] Lustre: Skipped 3 previous similar messages [ 7236.595365] Lustre: server umount lustre-MDT0000 complete [ 7238.461216] LustreError: 247688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835778 with bad export cookie 15219552699022713696 [ 7238.465482] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7238.473139] LustreError: 247688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7244.701910] Lustre: server umount lustre-MDT0001 complete [ 7250.975336] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7252.603990] Key type lgssc unregistered [ 7252.762625] LNet: 252812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7252.766767] LNetError: 252812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7252.778812] LNet: Removed LNI 192.168.202.159@tcp [ 7253.160809] Key type .llcrypt unregistered [ 7253.162528] Key type ._llcrypt unregistered [ 7277.721950] Key type ._llcrypt registered [ 7277.724730] Key type .llcrypt registered [ 7277.827513] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7278.669603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7278.692902] alg: No test for adler32 (adler32-zlib) [ 7279.648428] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7279.832358] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7281.479253] Key type lgssc registered [ 7282.307897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7286.973706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7286.991108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.115745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7288.126698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7288.215949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.706391] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7295.790195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7295.854584] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7295.876434] Lustre: 254996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7296.101210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7298.684301] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7303.269166] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7306.518535] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7310.529490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7310.655662] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7310.778272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.459821] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7318.585375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7319.798133] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7364.577696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7364.579235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.580110] LustreError: Skipped 1 previous similar message [ 7364.591864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7364.595342] Lustre: Skipped 1 previous similar message [ 7369.697943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7369.700666] Lustre: Skipped 1 previous similar message [ 7370.546929] Lustre: server umount lustre-OST0000 complete [ 7372.809380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7378.811468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.966134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.887345] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7385.854400] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7394.351359] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7394.362656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7394.369949] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7424.992645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7424.996300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.004870] Lustre: Skipped 1 previous similar message [ 7425.009537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7425.012268] Lustre: Skipped 1 previous similar message [ 7427.374906] Lustre: server umount lustre-OST0000 complete [ 7430.115990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7430.122793] Lustre: Skipped 1 previous similar message [ 7430.125467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7430.128673] Lustre: Skipped 1 previous similar message [ 7435.234538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7435.237646] Lustre: Skipped 1 previous similar message [ 7435.323849] Lustre: server umount lustre-MDT0000 complete [ 7437.706260] LustreError: 254988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835977 with bad export cookie 8212436708923453995 [ 7437.709811] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7437.716114] LustreError: 254988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7437.898043] Lustre: server umount lustre-MDT0001 complete [ 7443.390460] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7444.935339] Key type lgssc unregistered [ 7445.177541] LNet: 260416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7445.181610] LNetError: 260416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7445.191952] LNet: Removed LNI 192.168.202.159@tcp [ 7445.735665] Key type .llcrypt unregistered [ 7445.737036] Key type ._llcrypt unregistered [ 7459.847169] Key type ._llcrypt registered [ 7459.848514] Key type .llcrypt registered [ 7459.903599] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7460.668818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7460.685691] alg: No test for adler32 (adler32-zlib) [ 7461.603051] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7461.747134] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7463.359185] Key type lgssc registered [ 7464.005480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7468.518140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7468.531378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.947505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.629623] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7477.881851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.186656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7481.010157] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7486.603604] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.016191] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.038201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.231513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.305182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7498.890259] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7503.174446] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7517.152252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7517.156278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.163198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7519.206603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7519.207415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7519.207516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.225442] Lustre: Skipped 1 previous similar message [ 7521.753926] Lustre: server umount lustre-OST0000 complete [ 7524.323509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7524.324053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.325879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7524.350589] Lustre: Skipped 1 previous similar message [ 7529.441569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7529.445205] Lustre: Skipped 2 previous similar messages [ 7529.916989] Lustre: server umount lustre-MDT0000 complete [ 7531.947468] LustreError: 261798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836072 with bad export cookie 16598961975339626163 [ 7531.951905] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.955329] LustreError: 261798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7532.164042] Lustre: server umount lustre-MDT0001 complete [ 7537.713049] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7539.293734] Key type lgssc unregistered [ 7539.473056] LNet: 266125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7539.477136] LNetError: 266125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7539.490047] LNet: Removed LNI 192.168.202.159@tcp [ 7539.966934] Key type .llcrypt unregistered [ 7539.968474] Key type ._llcrypt unregistered [ 7550.511001] Key type ._llcrypt registered [ 7550.512898] Key type .llcrypt registered [ 7550.582707] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7551.266491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7551.285076] alg: No test for adler32 (adler32-zlib) [ 7552.200623] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7552.317093] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7553.951175] Key type lgssc registered [ 7554.731899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7560.018920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7560.034339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7561.371865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.586101] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7570.548789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.790116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7573.541368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7577.818317] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7581.262875] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7585.375919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7585.633808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.670952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7589.567947] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7594.307216] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7606.246151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.248655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7606.249128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7606.258348] Lustre: Skipped 1 previous similar message [ 7610.927738] Lustre: server umount lustre-OST0000 complete [ 7616.482405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.488323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7616.494703] Lustre: Skipped 2 previous similar messages [ 7621.601397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7621.604047] Lustre: Skipped 1 previous similar message [ 7626.720384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7626.726552] Lustre: Skipped 1 previous similar message [ 7627.231242] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7627.349367] Lustre: server umount lustre-MDT0000 complete [ 7629.654292] LustreError: 267304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836169 with bad export cookie 14054654364338842953 [ 7629.659494] LustreError: 267304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7629.661031] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7629.873591] Lustre: server umount lustre-MDT0001 complete [ 7635.676782] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7637.159247] Key type lgssc unregistered [ 7637.353547] LNet: 271004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7637.358455] LNetError: 271004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7637.367747] LNet: Removed LNI 192.168.202.159@tcp [ 7637.885204] Key type .llcrypt unregistered [ 7637.886749] Key type ._llcrypt unregistered [ 7650.649486] Key type ._llcrypt registered [ 7650.650582] Key type .llcrypt registered [ 7650.733708] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7651.216651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7651.233589] alg: No test for adler32 (adler32-zlib) [ 7652.164812] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7652.326343] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7653.943893] Key type lgssc registered [ 7654.491966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7658.913633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7658.926198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7660.146636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.947808] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7667.945876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.169609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7670.685259] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7674.999887] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7678.399948] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7682.915459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7683.234935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.311472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7686.984710] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7691.880037] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7709.165620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7709.170828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7709.176362] Lustre: Skipped 1 previous similar message [ 7709.176993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7709.182890] Lustre: Skipped 1 previous similar message [ 7714.277184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7714.281874] Lustre: Skipped 1 previous similar message [ 7714.602550] Lustre: server umount lustre-OST0000 complete [ 7719.394864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.394895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7719.400583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.407395] Lustre: Skipped 1 previous similar message [ 7722.766870] Lustre: server umount lustre-MDT0000 complete [ 7724.514380] LustreError: 273127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7724.520816] LustreError: 273127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7724.746585] LustreError: 272424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836264 with bad export cookie 260843086950090869 [ 7724.746864] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7724.754711] LustreError: 272424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7724.910982] Lustre: server umount lustre-MDT0001 complete [ 7730.058920] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7731.463943] Key type lgssc unregistered [ 7731.631674] LNet: 276702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7731.635190] LNetError: 276702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7731.648837] LNet: Removed LNI 192.168.202.159@tcp [ 7732.081538] Key type .llcrypt unregistered [ 7732.082928] Key type ._llcrypt unregistered [ 7741.998364] Key type ._llcrypt registered [ 7742.000414] Key type .llcrypt registered [ 7742.076398] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7742.610804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7742.633051] alg: No test for adler32 (adler32-zlib) [ 7743.547034] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7743.675457] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7745.287185] Key type lgssc registered [ 7745.806280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7750.219866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7750.235409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.495781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.669313] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7757.902561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.106744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7760.394678] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7764.033699] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.853319] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7770.188984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7770.366830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.838162] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7776.423870] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7776.554677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7792.096290] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7792.099078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.106598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.145059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.146991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7794.153845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7797.480152] Lustre: server umount lustre-OST0000 complete [ 7799.265287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.265446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7799.274687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.284786] Lustre: Skipped 2 previous similar messages [ 7804.384372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7804.386862] Lustre: Skipped 1 previous similar message [ 7805.426375] Lustre: server umount lustre-MDT0000 complete [ 7807.651307] LustreError: 277893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836347 with bad export cookie 303709262004049953 [ 7807.655269] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7807.661640] LustreError: 277893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7807.899140] Lustre: server umount lustre-MDT0001 complete [ 7813.774894] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7815.547584] Key type lgssc unregistered [ 7815.718484] LNet: 281644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7815.721838] LNetError: 281644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7815.728448] LNet: Removed LNI 192.168.202.159@tcp [ 7816.318356] Key type .llcrypt unregistered [ 7816.319794] Key type ._llcrypt unregistered [ 7821.509786] Key type ._llcrypt registered [ 7821.511274] Key type .llcrypt registered [ 7821.571348] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7830.786671] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7838.913612] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7839.487460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7839.502048] alg: No test for adler32 (adler32-zlib) [ 7840.412869] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 7840.532254] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7842.159230] Key type lgssc registered [ 7842.741405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7861.294253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7861.304316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7862.442832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7862.460246] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7862.530381] Lustre: lustre-MDT0000: new disk, initializing [ 7862.580378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7862.595139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7864.723915] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7871.291683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.381236] Lustre: 285170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7871.459272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7871.465140] Lustre: Skipped 1 previous similar message [ 7871.539541] Lustre: lustre-MDT0001: new disk, initializing [ 7871.613519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7871.638174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7871.643217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7874.214715] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7879.611578] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7882.518444] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7886.370277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7886.541526] Lustre: lustre-OST0000: new disk, initializing [ 7886.544596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7886.549144] Lustre: 287137:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7886.621187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.545738] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7894.022615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7894.026886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7894.105607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7894.855822] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7896.853253] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7896.959309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7898.824093] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7898.923385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7904.224080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7904.227453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.229480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7904.234882] Lustre: Skipped 1 previous similar message [ 7905.969562] Lustre: server umount lustre-OST0000 complete [ 7909.346052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.351652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7909.354731] Lustre: Skipped 2 previous similar messages [ 7913.829716] Lustre: server umount lustre-MDT0000 complete [ 7915.260087] LustreError: 285163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836455 with bad export cookie 4604155460269451153 [ 7915.263021] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7915.263914] LustreError: 285163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7915.399402] Lustre: server umount lustre-MDT0001 complete [ 7919.012364] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:27:39 (1786836459) [ 7921.931176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7922.118377] LustreError: 289546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7922.126830] LustreError: 289546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7922.146748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.763438] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7927.179574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.267531] LustreError: 289547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7927.369660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7928.976778] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7932.290913] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7935.065492] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7938.344066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.522813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.059338] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 7944.180063] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7959.008797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7959.018528] LustreError: Skipped 1 previous similar message [ 7959.019940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.026128] Lustre: Skipped 1 previous similar message [ 7959.029390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7963.105860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.106786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7963.117117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.228154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7968.232085] Lustre: Skipped 1 previous similar message [ 7972.319132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7972.392294] Lustre: server umount lustre-OST0000 complete [ 7978.465748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.465878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7978.466737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7978.466742] Lustre: Skipped 1 previous similar message [ 7978.470616] Lustre: Skipped 1 previous similar message [ 7980.263857] Lustre: server umount lustre-MDT0000 complete [ 7982.081322] LustreError: 289528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836522 with bad export cookie 4604155460269452035 [ 7982.084296] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7982.085668] LustreError: 289528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7982.251798] Lustre: server umount lustre-MDT0001 complete [ 7987.234631] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7988.675194] Key type lgssc unregistered [ 7988.851563] LNet: 293104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7988.854382] LNetError: 293104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7988.862418] LNet: Removed LNI 192.168.202.159@tcp [ 7989.234945] Key type .llcrypt unregistered [ 7989.236798] Key type ._llcrypt unregistered [ 8002.361431] Key type ._llcrypt registered [ 8002.362996] Key type .llcrypt registered [ 8002.411019] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8010.089988] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8010.478463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8010.536029] alg: No test for adler32 (adler32-zlib) [ 8011.418901] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8011.533615] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8013.119174] Key type lgssc registered [ 8013.602582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8034.489811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8034.505305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8035.641200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8035.651917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8035.691273] Lustre: lustre-MDT0000: new disk, initializing [ 8035.735452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.746585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8037.443875] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8043.020890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8043.080160] Lustre: 296904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8043.109833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8043.112613] Lustre: Skipped 1 previous similar message [ 8043.162540] Lustre: lustre-MDT0001: new disk, initializing [ 8043.209844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8043.225214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8043.228343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8044.966911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8049.604059] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8052.105865] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8055.304880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.446261] Lustre: lustre-OST2710: new disk, initializing [ 8055.448397] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8055.451818] Lustre: 298872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8055.508948] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8058.310246] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8061.444655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8061.448358] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8061.488529] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8062.768102] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.838415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8065.933121] Lustre: lustre-OST03e8: new disk, initializing [ 8065.935258] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8065.944080] Lustre: 299935:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8065.995277] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8066.778769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8066.785936] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8066.822974] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8068.694445] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8072.316066] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8075.699346] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8075.797856] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8077.558474] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8077.648206] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8079.481995] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8079.576745] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8081.394867] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8081.483059] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8084.448494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8084.452042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.459059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.522731] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8087.522833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.527255] Lustre: Skipped 1 previous similar message [ 8087.529517] Lustre: Skipped 1 previous similar message [ 8089.165397] Lustre: server umount lustre-MDT0000 complete [ 8091.041343] LustreError: 296897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836631 with bad export cookie 9422921276378427237 [ 8091.045715] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8091.051287] LustreError: 296897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8091.313859] Lustre: server umount lustre-MDT0001 complete [ 8103.588623] Lustre: server umount lustre-OST2710 complete [ 8115.900061] Lustre: server umount lustre-OST03e8 complete [ 8125.262468] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8129.649415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8146.111172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.237010] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8146.256919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8146.299870] Lustre: lustre-MDT0000: new disk, initializing [ 8146.342943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.352945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8148.299787] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8153.517130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.569875] Lustre: 305180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8153.688797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8155.669445] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8160.574154] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8163.420574] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8166.964280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.158882] Lustre: lustre-OST0000: new disk, initializing [ 8167.164052] Lustre: Skipped 1 previous similar message [ 8167.167750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8167.172557] Lustre: Skipped 2 previous similar messages [ 8167.176135] Lustre: 307146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8167.231374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8167.237049] Lustre: Skipped 1 previous similar message [ 8167.908146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8167.913741] Lustre: Skipped 1 previous similar message [ 8167.918790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8167.954037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8170.109928] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8175.208505] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8177.414890] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8177.539482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8179.668944] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8179.796993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8183.270082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8183.275368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.282643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.284745] Lustre: Skipped 2 previous similar messages [ 8183.290640] Lustre: Skipped 3 previous similar messages [ 8188.386189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8188.389898] Lustre: Skipped 1 previous similar message [ 8193.506940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8193.511100] Lustre: Skipped 1 previous similar message [ 8195.042801] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8195.113707] Lustre: server umount lustre-OST0000 complete [ 8198.625463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.721556] Lustre: server umount lustre-MDT0000 complete [ 8204.387305] LustreError: 305171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836744 with bad export cookie 9422921276378428903 [ 8204.390595] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8204.392157] LustreError: 305171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8204.554980] Lustre: server umount lustre-MDT0001 complete [ 8208.014417] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:32:28 (1786836748) [ 8208.683586] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8209.415578] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:32:30 (1786836750) [ 8210.671549] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8222.159400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8222.268117] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8222.282639] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8222.366977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8224.242897] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8228.969272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.032474] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8229.053533] Lustre: 310995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8231.879126] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8249.827623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8249.916095] Lustre: Denying initial registration attempt from nid 192.168.202.159@tcp, specified as failover [ 8249.922661] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8249.927082] LustreError: 312448:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8249.930409] LustreError: 312448:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8249.933145] LustreError: 312448:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8249.971270] Lustre: server umount lustre-OST0000 complete [ 8249.973140] LustreError: 312448:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8254.438934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.439772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.447360] Lustre: Skipped 2 previous similar messages [ 8254.452957] Lustre: Skipped 3 previous similar messages [ 8259.274348] Lustre: server umount lustre-MDT0000 complete [ 8259.552391] LustreError: 311008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8259.562354] LustreError: 311008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8261.300043] LustreError: 310990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836801 with bad export cookie 9422921276378429785 [ 8261.301275] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8261.308571] LustreError: 310990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8261.491495] Lustre: server umount lustre-MDT0001 complete [ 8266.517079] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8267.740174] Key type lgssc unregistered [ 8267.880424] LNet: 313571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8267.883104] LNetError: 313571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8267.894349] LNet: Removed LNI 192.168.202.159@tcp [ 8268.226079] Key type .llcrypt unregistered [ 8268.227836] Key type ._llcrypt unregistered [ 8273.321037] Key type ._llcrypt registered [ 8273.322057] Key type .llcrypt registered [ 8273.379507] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8273.935931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8273.942729] alg: No test for adler32 (adler32-zlib) [ 8274.839596] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8274.959699] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8276.552111] Key type lgssc registered [ 8277.198564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.160331] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8308.468392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8308.480126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.626348] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8309.638284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8309.725587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.885358] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8316.238423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8316.314941] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8316.329566] Lustre: 316311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8316.470561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8318.511374] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8322.106142] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8324.582870] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8340.898608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8340.985198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8341.063861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8343.474628] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8346.758058] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8347.719340] Lustre: server umount lustre-OST0000 complete [ 8352.228797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8352.234065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8352.246146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.248613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.249549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.357166] Lustre: server umount lustre-MDT0000 complete [ 8356.859552] LustreError: 316303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836896 with bad export cookie 2433667438756441037 [ 8356.863946] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8356.864819] LustreError: 316303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8357.013053] Lustre: server umount lustre-MDT0001 complete [ 8361.354331] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8362.469925] Key type lgssc unregistered [ 8362.624486] LNet: 319879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8362.627191] LNetError: 319879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8362.633362] LNet: Removed LNI 192.168.202.159@tcp [ 8362.974598] Key type .llcrypt unregistered [ 8362.976077] Key type ._llcrypt unregistered [ 8373.150445] Key type ._llcrypt registered [ 8373.151848] Key type .llcrypt registered [ 8373.189157] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8373.668773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8373.682536] alg: No test for adler32 (adler32-zlib) [ 8374.578410] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8374.687219] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8376.287225] Key type lgssc registered [ 8376.758240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8380.152505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8380.174899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.416816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8383.214965] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8387.224073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.402495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8389.273464] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8392.642404] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8395.149072] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8400.035204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8400.189111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8402.426141] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8405.537098] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8422.137117] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 8422.889646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.895741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8422.898659] Lustre: Skipped 2 previous similar messages [ 8422.903185] Lustre: Skipped 2 previous similar messages [ 8425.883931] Lustre: server umount lustre-MDT0000 complete [ 8427.251072] LustreError: 321270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.263442] LustreError: 321270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8427.472356] LustreError: 323117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836967 with bad export cookie 9125492984914739999 [ 8427.477316] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8427.477695] LustreError: 323117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8428.000346] LustreError: 321269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8428.000817] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8428.013659] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8432.368583] LustreError: 321268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8432.377925] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 8433.688693] Lustre: server umount lustre-MDT0001 complete [ 8436.416667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8440.214050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.575781] LustreError: 320583:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7d2a55d x1873634303961472/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8452.735230] LustreError: 323103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.744581] LustreError: 323103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8452.766864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8452.798187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 8454.498895] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8457.842601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8457.890559] LustreError: 323104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8457.901019] LustreError: 323104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8458.054216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8459.774587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8462.181660] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8462.988765] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8468.735715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 8470.894484] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8473.545782] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 8477.665830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8477.667782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8477.669995] Lustre: Skipped 1 previous similar message [ 8477.672799] Lustre: Skipped 1 previous similar message [ 8481.179615] Lustre: server umount lustre-OST0000 complete [ 8482.784544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.789540] Lustre: Skipped 1 previous similar message [ 8487.905191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.908988] Lustre: Skipped 3 previous similar messages [ 8488.740287] Lustre: server umount lustre-MDT0000 complete [ 8490.179184] LustreError: 321253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837030 with bad export cookie 9125492984914747153 [ 8490.183063] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8490.183946] LustreError: 321253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8490.299507] Lustre: server umount lustre-MDT0001 complete [ 8494.403184] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8495.632501] Key type lgssc unregistered [ 8495.772383] LNet: 327558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8495.775172] LNetError: 327558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8495.782327] LNet: Removed LNI 192.168.202.159@tcp [ 8496.139151] Key type .llcrypt unregistered [ 8496.141374] Key type ._llcrypt unregistered [ 8506.329287] Key type ._llcrypt registered [ 8506.330725] Key type .llcrypt registered [ 8506.372298] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8506.695316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8506.702598] alg: No test for adler32 (adler32-zlib) [ 8507.577415] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8507.669788] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8509.255155] Key type lgssc registered [ 8509.674894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8512.415152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8512.421990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.592248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.176228] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8518.504216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8518.682387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8520.221052] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8523.145560] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8525.330214] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8528.481795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.487572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8528.490442] Lustre: Skipped 1 previous similar message [ 8532.566370] Lustre: server umount lustre-MDT0000 complete [ 8533.741186] LustreError: 328933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837073 with bad export cookie 8414490476321449997 [ 8533.744363] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8533.745919] LustreError: 328933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8533.854210] Lustre: server umount lustre-MDT0001 complete [ 8538.603834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8538.696903] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8538.710112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8538.782049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.150431] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8543.047418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.081137] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8543.091824] Lustre: 331536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8543.197035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8544.581834] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8547.099695] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8549.089515] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8553.957618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8554.034545] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8554.038432] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8554.042356] LustreError: 333417:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 8554.045158] LustreError: 333417:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8554.047591] LustreError: 333417:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8554.072926] Lustre: server umount lustre-OST0000 complete [ 8554.074480] LustreError: 333417:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8559.265386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8559.328893] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8559.393120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8561.389298] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8563.814651] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8566.350284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8566.382675] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8566.405384] Lustre: lustre-OST0001: new disk, initializing [ 8566.407846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8566.411454] Lustre: 334948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8568.277565] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8568.904979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8568.908552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8568.924947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8568.952892] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8570.592090] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8573.920282] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8573.922431] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.925679] Lustre: Skipped 1 previous similar message [ 8573.927500] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8574.434898] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8577.602455] Lustre: server umount lustre-OST0001 complete [ 8579.040478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8579.043408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8579.047730] Lustre: Skipped 1 previous similar message [ 8579.051322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8579.054575] Lustre: Skipped 2 previous similar messages [ 8579.554317] LustreError: 333884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8579.559787] LustreError: 333884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8584.673620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8584.674032] LustreError: 333884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8584.675066] Lustre: Skipped 2 previous similar messages [ 8584.679033] LustreError: 333884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8589.793493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8589.794249] LustreError: 333883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.795478] Lustre: Skipped 1 previous similar message [ 8589.798994] LustreError: 333883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8593.375179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8593.425170] Lustre: server umount lustre-OST0000 complete [ 8594.912232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.914767] Lustre: Skipped 1 previous similar message [ 8599.519983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8599.522122] LustreError: Skipped 1 previous similar message [ 8599.523145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.526875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8599.528260] Lustre: Skipped 1 previous similar message [ 8600.792608] Lustre: server umount lustre-MDT0000 complete [ 8601.993282] LustreError: 331528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837142 with bad export cookie 8414490476321450536 [ 8601.996868] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8601.997870] LustreError: 331528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8602.110442] Lustre: server umount lustre-MDT0001 complete [ 8605.830142] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8606.871100] Key type lgssc unregistered [ 8606.988377] LNet: 336909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8606.990725] LNetError: 336909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8606.997249] LNet: Removed LNI 192.168.202.159@tcp [ 8607.314944] Key type .llcrypt unregistered [ 8607.316026] Key type ._llcrypt unregistered [ 8638.949790] Key type ._llcrypt registered [ 8638.950641] Key type .llcrypt registered [ 8638.991437] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 8644.966076] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8645.421919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8645.435635] alg: No test for adler32 (adler32-zlib) [ 8646.293204] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8646.387681] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8647.975168] Key type lgssc registered [ 8648.338748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8660.904409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8660.911971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.998779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8662.011104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8662.050277] Lustre: lustre-MDT0000: new disk, initializing [ 8662.071139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8662.077884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8663.439029] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8667.687499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.738253] Lustre: 341811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8667.763560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8667.766136] Lustre: Skipped 1 previous similar message [ 8667.802058] Lustre: lustre-MDT0001: new disk, initializing [ 8667.834317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8667.847677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8667.851547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8669.301468] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8672.956709] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8674.689258] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8676.933859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8677.037142] Lustre: lustre-OST0000: new disk, initializing [ 8677.038844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8677.041896] Lustre: 343781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8677.070663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.873721] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8679.035115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8679.039082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8679.072906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8682.523413] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8683.902024] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8683.969071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8685.336342] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8685.405131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8689.632408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8689.633303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.634561] LustreError: Skipped 1 previous similar message [ 8689.640637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8689.643745] Lustre: Skipped 1 previous similar message [ 8694.753986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8694.756534] Lustre: Skipped 1 previous similar message [ 8699.873866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8699.876662] Lustre: Skipped 1 previous similar message [ 8700.383093] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8700.438549] Lustre: server umount lustre-OST0000 complete [ 8703.967831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8703.970233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.974068] Lustre: Skipped 1 previous similar message [ 8703.975970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8704.994340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.873955] Lustre: server umount lustre-MDT0000 complete [ 8709.391676] LustreError: 342608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837249 with bad export cookie 5420552386694648692 [ 8709.395221] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8709.398269] LustreError: 342608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8709.533850] Lustre: server umount lustre-MDT0001 complete [ 8712.647367] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:40:53 (1786837253) [ 8715.156250] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:40:55 (1786837255) [ 8718.284731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8718.518231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8719.982383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8723.016031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.141162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8724.579209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8727.044433] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8728.847408] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8731.083208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.213100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8733.203929] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8735.672283] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8750.531120] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8751.041592] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8751.519492] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8752.019410] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8752.610052] Lustre: Failing over lustre-MDT0000 [ 8752.774966] Lustre: server umount lustre-MDT0000 complete [ 8754.145594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.146715] LustreError: 348788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.148980] Lustre: Skipped 2 previous similar messages [ 8754.153330] LustreError: 348788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8754.155411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8757.495322] LustreError: 346530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.264441] LustreError: 346536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8759.268693] LustreError: 346536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8762.608210] LustreError: 346532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8765.463610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.499318] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.583461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.604960] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8766.759377] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8767.727430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8771.044205] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8771.053959] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8771.068496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8772.774926] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8773.238501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8775.029752] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8775.643930] Lustre: Failing over lustre-MDT0000 [ 8775.819843] Lustre: server umount lustre-MDT0000 complete [ 8776.162191] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.162598] LustreError: 348788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.162986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8776.165239] Lustre: Skipped 2 previous similar messages [ 8776.172333] LustreError: 348788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 8786.399988] LustreError: 348788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.403454] LustreError: 348788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8788.585741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.625376] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8788.752429] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8789.979966] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8793.327604] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8794.087220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8794.088919] Lustre: Skipped 2 previous similar messages [ 8794.099086] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8794.115410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8795.850632] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8796.330816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8799.201451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.202989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8799.205647] Lustre: Skipped 2 previous similar messages [ 8799.205909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8799.208154] Lustre: Skipped 4 previous similar messages [ 8804.632440] Lustre: server umount lustre-MDT0000 complete [ 8805.834051] LustreError: 349326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837345 with bad export cookie 5420552386694651373 [ 8805.836787] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8805.838446] LustreError: 349326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8805.944795] Lustre: server umount lustre-MDT0001 complete [ 8817.387716] Lustre: server umount lustre-OST0000 complete [ 8824.568639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8824.766352] LustreError: 352886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8824.773435] LustreError: 352886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8824.796340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8824.799257] Lustre: Skipped 1 previous similar message [ 8826.274101] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8829.196317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.643725] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8833.037742] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8834.955946] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8837.248027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8839.390865] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8842.009449] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8849.639256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8855.010707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8855.010890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.011556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8855.011560] Lustre: Skipped 3 previous similar messages [ 8855.019147] Lustre: Skipped 1 previous similar message [ 8856.018864] Lustre: server umount lustre-OST0000 complete [ 8859.478091] Lustre: server umount lustre-MDT0000 complete [ 8860.130034] LustreError: 352911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8860.137174] LustreError: 352911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8860.806413] LustreError: 352866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837400 with bad export cookie 5420552386694652080 [ 8860.810068] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8860.811899] LustreError: 352866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8860.922496] Lustre: server umount lustre-MDT0001 complete [ 8864.785951] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 8865.836411] Key type lgssc unregistered [ 8865.974342] LNet: 356437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8865.977458] LNetError: 356437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8865.990335] LNet: Removed LNI 192.168.202.159@tcp [ 8866.317393] Key type .llcrypt unregistered [ 8866.318583] Key type ._llcrypt unregistered [ 8876.211773] Key type ._llcrypt registered [ 8876.212978] Key type .llcrypt registered [ 8876.249928] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 8876.678637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8876.686122] alg: No test for adler32 (adler32-zlib) [ 8877.528797] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 8877.610610] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 8879.191124] Key type lgssc registered [ 8879.517387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8881.882570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8881.888582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.020989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8884.357415] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8887.048870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.164501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8888.508809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8890.873312] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8892.665883] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8894.771859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8894.882498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.796326] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8899.200604] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8908.198734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8917.984175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8917.986271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8917.989443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8918.497314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8918.497733] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.575770] Lustre: server umount lustre-MDT0000 complete [ 8920.790136] LustreError: 357910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837460 with bad export cookie 16757039866252115956 [ 8920.793526] LustreError: 357910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8920.794190] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8920.909182] Lustre: server umount lustre-MDT0001 complete [ 8932.438949] Lustre: server umount lustre-OST0000 complete [ 8936.944297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.111870] LustreError: 361657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.117479] LustreError: 361657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8937.140209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.558629] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8941.516975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.646284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8943.087898] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8945.647697] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8947.563236] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8949.863200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.981499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8952.011635] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8954.535081] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8962.279527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8972.255840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8972.258189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.261791] Lustre: Skipped 1 previous similar message [ 8972.263677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8972.266443] Lustre: Skipped 2 previous similar messages [ 8977.807833] Lustre: server umount lustre-MDT0000 complete [ 8977.888974] LustreError: 361658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8977.894533] LustreError: 361658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8978.932903] LustreError: 364271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837519 with bad export cookie 16757039866252117055 [ 8978.935765] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8978.938038] LustreError: 364271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8979.040875] Lustre: server umount lustre-MDT0001 complete [ 8990.426210] Lustre: server umount lustre-OST0000 complete [ 8995.475050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8995.621964] LustreError: 365585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.642895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8996.910357] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 8999.590228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.944049] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9003.216099] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.045333] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9007.209278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.143984] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9011.418413] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9021.669776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9024.991799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9024.994032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9024.997376] Lustre: Skipped 2 previous similar messages [ 9024.998997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9025.000832] Lustre: Skipped 2 previous similar messages [ 9027.042303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9028.943674] Lustre: server umount lustre-OST0000 complete [ 9030.623849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.628390] Lustre: Skipped 1 previous similar message [ 9030.630510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.632802] Lustre: Skipped 2 previous similar messages [ 9036.315247] Lustre: server umount lustre-MDT0000 complete [ 9037.280221] LustreError: 365604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.286673] LustreError: 365604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9037.541105] LustreError: 368186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837577 with bad export cookie 16757039866252118063 [ 9037.544607] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9037.545350] LustreError: 368186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9037.666361] Lustre: server umount lustre-MDT0001 complete [ 9041.578714] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9042.582042] Key type lgssc unregistered [ 9042.713544] LNet: 369141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9042.715950] LNetError: 369141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9042.723287] LNet: Removed LNI 192.168.202.159@tcp [ 9043.050578] Key type .llcrypt unregistered [ 9043.051899] Key type ._llcrypt unregistered [ 9052.671391] Key type ._llcrypt registered [ 9052.672588] Key type .llcrypt registered [ 9052.718847] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9053.050491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9053.080123] alg: No test for adler32 (adler32-zlib) [ 9053.949922] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9054.060700] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9055.647154] Key type lgssc registered [ 9056.023633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9058.500874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9058.506936] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9058.510028] LustreError: 370557:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9058.519522] LustreError: 370557:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9058.522120] LustreError: 370557:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9058.524761] LustreError: 370557:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9058.527098] LustreError: 370557:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9061.070760] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.073246] LustreError: 370956:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9061.081864] LustreError: 370956:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9061.084266] LustreError: 370956:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9061.087066] LustreError: 370956:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9061.089310] LustreError: 370956:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9067.776976] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9068.793315] Key type lgssc unregistered [ 9068.924320] LNet: 371959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9068.926902] LNetError: 371959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9068.936267] LNet: Removed LNI 192.168.202.159@tcp [ 9069.234356] Key type .llcrypt unregistered [ 9069.235975] Key type ._llcrypt unregistered [ 9075.139031] Key type ._llcrypt registered [ 9075.140320] Key type .llcrypt registered [ 9075.177374] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9081.612883] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9082.056165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9082.077129] alg: No test for adler32 (adler32-zlib) [ 9082.936792] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9083.026036] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9084.615158] Key type lgssc registered [ 9084.967429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9096.781856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9096.788191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.869140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9097.879322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9097.918642] Lustre: lustre-MDT0000: new disk, initializing [ 9097.947441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.954402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9099.255262] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9103.296025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.339154] Lustre: 375086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9103.361789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9103.363405] Lustre: Skipped 1 previous similar message [ 9103.396807] Lustre: lustre-MDT0001: new disk, initializing [ 9103.428226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9103.441806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9103.446312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9104.681630] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9108.397495] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9110.338103] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9112.563641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9112.664061] Lustre: lustre-OST0000: new disk, initializing [ 9112.665829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9112.668688] Lustre: 377054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9112.699916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.672840] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9116.281459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9116.286665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9116.319932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9118.473650] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9119.915167] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9119.990020] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9121.302654] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9121.366607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9126.880630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9126.882174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.883288] LustreError: Skipped 1 previous similar message [ 9126.888310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9126.892165] Lustre: Skipped 1 previous similar message [ 9128.148444] Lustre: server umount lustre-OST0000 complete [ 9132.000694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.001434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9132.003781] Lustre: Skipped 2 previous similar messages [ 9135.511533] Lustre: server umount lustre-MDT0000 complete [ 9136.670078] LustreError: 375079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837676 with bad export cookie 12761704851414330135 [ 9136.673092] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9136.673328] LustreError: 375079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9136.783310] Lustre: server umount lustre-MDT0001 complete [ 9139.454254] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9139.958913] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:48:00 (1786837680) [ 9142.163380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.336177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9143.559942] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9146.280127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.405685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.726717] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9150.095165] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9151.763160] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9153.937557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9155.870816] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9158.177964] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9160.333553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9160.392140] Lustre: lustre-OST0001: new disk, initializing [ 9160.393646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9160.395798] Lustre: 382343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9160.422746] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9160.425101] Lustre: Skipped 1 previous similar message [ 9162.189994] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9165.686579] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9169.407044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9169.410250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9169.443226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9174.497694] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9174.499712] LustreError: Skipped 1 previous similar message [ 9174.499721] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9174.502170] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9174.506737] Lustre: Skipped 1 previous similar message [ 9174.508086] Lustre: Skipped 1 previous similar message [ 9177.984363] Lustre: server umount lustre-OST0001 complete [ 9179.619984] LustreError: 381333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.625637] LustreError: 381333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9184.738871] LustreError: 381330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9184.743170] LustreError: 381330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9189.859581] LustreError: 381333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.865274] LustreError: 381333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9191.904016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9191.907566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9191.913425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9191.915086] Lustre: Skipped 2 previous similar messages [ 9194.982996] LustreError: 381330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9194.990980] LustreError: 381330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9196.243118] Lustre: server umount lustre-OST0000 complete [ 9200.096997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.101089] Lustre: Skipped 1 previous similar message [ 9200.103433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9200.106565] Lustre: Skipped 4 previous similar messages [ 9203.753682] Lustre: server umount lustre-MDT0000 complete [ 9205.108307] LustreError: 382342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837745 with bad export cookie 12761704851414331003 [ 9205.111081] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.111528] LustreError: 382342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9211.323568] Lustre: server umount lustre-MDT0001 complete [ 9215.193934] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9216.237850] Key type lgssc unregistered [ 9216.359404] LNet: 384332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9216.361785] LNetError: 384332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9216.369324] LNet: Removed LNI 192.168.202.159@tcp [ 9216.692775] Key type .llcrypt unregistered [ 9216.693894] Key type ._llcrypt unregistered [ 9235.555559] Key type ._llcrypt registered [ 9235.556627] Key type .llcrypt registered [ 9235.593973] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9235.925568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9235.933729] alg: No test for adler32 (adler32-zlib) [ 9236.797959] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9236.890510] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9238.471150] Key type lgssc registered [ 9238.846835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9241.546463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9241.554505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9242.644627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9242.654145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9242.713334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.218929] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9247.150767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.186725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9247.197067] Lustre: 387115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9247.314691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.728673] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9251.346818] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9253.298862] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9256.928222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.929237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9256.931772] Lustre: Skipped 1 previous similar message [ 9262.048151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9262.050055] Lustre: Skipped 2 previous similar messages [ 9267.168282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9267.171278] Lustre: Skipped 1 previous similar message [ 9268.191160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9268.254241] Lustre: server umount lustre-MDT0000 complete [ 9269.567512] LustreError: 387857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837809 with bad export cookie 15003968653827412233 [ 9269.571561] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9269.573674] LustreError: 387857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9269.686668] Lustre: server umount lustre-MDT0001 complete [ 9272.995502] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:50:13 (1786837813) [ 9275.261184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.447601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.818593] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9279.799595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.929199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9281.346132] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9284.131421] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9286.134818] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9288.604751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9288.677912] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9288.747555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9290.840757] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9293.437289] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9297.182027] Lustre: Permanently deactivating lustre-OST0000 [ 9297.186726] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9306.094058] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9306.884150] LustreError: 392753:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9306.887520] LustreError: 392753:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9306.889492] LustreError: 392753:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9307.524161] LustreError: 392801:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9307.526712] LustreError: 392801:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9314.255838] Lustre: server umount lustre-OST0000 complete [ 9315.227583] LustreError: 393051:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9315.229921] LustreError: 393051:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9320.416702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.417394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9320.422810] Lustre: Skipped 1 previous similar message [ 9325.536137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.538262] Lustre: Skipped 2 previous similar messages [ 9326.585154] Lustre: server umount lustre-MDT0000 complete [ 9328.112212] LustreError: 392313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837868 with bad export cookie 15003968653827412800 [ 9328.115095] LustreError: 392313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9328.115284] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.389696] Lustre: server umount lustre-MDT0001 complete [ 9341.459033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9342.810852] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9343.674719] LustreError: 394208:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9345.718504] Lustre: 394547:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.159@tcp [ 9346.386528] Lustre: server umount MGS complete [ 9349.853860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.989755] LustreError: 395269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9349.993733] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9349.994546] LustreError: 395269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9349.995923] Lustre: Skipped 1 previous similar message [ 9350.011664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.202736] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9353.823093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.903306] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9355.143144] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9357.441035] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9359.271551] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9359.890948] Lustre: Permanently reactivating lustre-OST0000 [ 9359.896608] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9359.899561] Lustre: Skipped 2 previous similar messages [ 9381.343135] LustreError: 396907:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9384.162417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9384.266105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.268202] Lustre: Skipped 1 previous similar message [ 9386.137673] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9388.466452] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9401.825723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9401.825802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9401.828933] LustreError: Skipped 1 previous similar message [ 9401.829532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9401.832703] Lustre: Skipped 1 previous similar message [ 9401.837257] Lustre: Skipped 1 previous similar message [ 9402.451565] Lustre: server umount lustre-OST0000 complete [ 9405.407981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9405.410364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9405.846032] Lustre: server umount lustre-MDT0000 complete [ 9406.944593] LustreError: 395288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9406.949562] LustreError: 395288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9407.070286] LustreError: 395249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837947 with bad export cookie 15003968653827413836 [ 9407.071498] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9407.073511] LustreError: 395249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9407.078599] Lustre: 395261:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 9407.197132] Lustre: server umount lustre-MDT0001 complete [ 9411.047936] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9412.149147] Key type lgssc unregistered [ 9412.277394] LNet: 399798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9412.281252] LNetError: 399798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9412.289321] LNet: Removed LNI 192.168.202.159@tcp [ 9412.602358] Key type .llcrypt unregistered [ 9412.603492] Key type ._llcrypt unregistered [ 9418.702365] Key type ._llcrypt registered [ 9418.703265] Key type .llcrypt registered [ 9418.743533] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9425.094256] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9425.402384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9425.416387] alg: No test for adler32 (adler32-zlib) [ 9426.296920] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9426.383474] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9427.967180] Key type lgssc registered [ 9428.374884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9440.066722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9440.071632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9441.155715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9441.165100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9441.202338] Lustre: lustre-MDT0000: new disk, initializing [ 9441.222587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9441.228569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9442.580250] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9446.721297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9446.760560] Lustre: 402926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9446.782744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9446.784569] Lustre: Skipped 1 previous similar message [ 9446.825615] Lustre: lustre-MDT0001: new disk, initializing [ 9446.858996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9446.870755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9446.873682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9448.229348] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9451.963506] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9453.733155] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9456.019789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9456.134560] Lustre: lustre-OST0000: new disk, initializing [ 9456.137509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9456.141719] Lustre: 404894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9456.171660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9458.117239] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9458.488151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9458.492428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9458.539071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9461.860766] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9463.300410] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9463.378154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9464.792731] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9464.864643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9468.900417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.901092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9468.905580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9468.906830] LustreError: Skipped 1 previous similar message [ 9468.911672] Lustre: Skipped 1 previous similar message [ 9471.638802] Lustre: server umount lustre-OST0000 complete [ 9474.016913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9474.017608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9474.020440] Lustre: Skipped 2 previous similar messages [ 9479.136520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9479.138713] Lustre: Skipped 2 previous similar messages [ 9484.256738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9484.259947] Lustre: Skipped 1 previous similar message [ 9487.327158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9487.390969] Lustre: server umount lustre-MDT0000 complete [ 9488.671363] LustreError: 402919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838028 with bad export cookie 6670879913369265318 [ 9488.675604] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9488.676331] LustreError: 402919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9488.819104] Lustre: server umount lustre-MDT0001 complete [ 9491.657549] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9492.270296] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:53:52 (1786838032) [ 9494.583383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.762317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.048670] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9498.853859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.980667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9500.299955] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9502.580402] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9504.414723] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9506.639744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.700736] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9511.100544] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9511.756130] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9522.506398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9524.194695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9524.196699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.198583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9524.199814] Lustre: Skipped 1 previous similar message [ 9530.061446] Lustre: server umount lustre-OST0000 complete [ 9534.432634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9534.437574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9534.439442] Lustre: Skipped 4 previous similar messages [ 9537.369415] Lustre: server umount lustre-MDT0000 complete [ 9538.578173] LustreError: 407332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838078 with bad export cookie 6670879913369266186 [ 9538.580768] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9538.581261] LustreError: 407332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9538.704441] Lustre: server umount lustre-MDT0001 complete [ 9542.227438] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9543.221821] Key type lgssc unregistered [ 9543.348394] LNet: 411096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9543.350611] LNetError: 411096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9543.359276] LNet: Removed LNI 192.168.202.159@tcp [ 9543.653448] Key type .llcrypt unregistered [ 9543.654530] Key type ._llcrypt unregistered [ 9551.254100] Key type ._llcrypt registered [ 9551.255094] Key type .llcrypt registered [ 9551.290400] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9551.393328] Key type .llcrypt unregistered [ 9551.394324] Key type ._llcrypt unregistered [ 9558.487119] Key type ._llcrypt registered [ 9558.488272] Key type .llcrypt registered [ 9558.524324] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9558.936145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9558.952910] alg: No test for adler32 (adler32-zlib) [ 9559.812681] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9559.897772] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9561.479123] Key type lgssc registered [ 9561.818898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9564.277372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9564.284279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.424988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9566.767667] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9569.513641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9569.625261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9571.504966] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9573.881388] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9574.880283] LustreError: 413288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.887580] LustreError: 413288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9575.905429] LustreError: 413287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9575.909456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9575.911084] LustreError: 413287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9576.091441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.218591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9577.548968] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9586.143871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9586.146099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9586.150049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9586.657230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9586.659576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9586.664687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9590.603465] Lustre: server umount lustre-OST0000 complete [ 9594.850707] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9594.853364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9594.857129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.896131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.896680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9597.980891] Lustre: server umount lustre-MDT0000 complete [ 9599.229336] LustreError: 413270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838139 with bad export cookie 5630496230144061589 [ 9599.232234] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9599.232916] LustreError: 413270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9599.356094] Lustre: server umount lustre-MDT0001 complete [ 9603.115730] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9604.229140] Key type lgssc unregistered [ 9604.359436] LNet: 416437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9604.362342] LNetError: 416437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9604.371403] LNet: Removed LNI 192.168.202.159@tcp [ 9604.690073] Key type .llcrypt unregistered [ 9604.691209] Key type ._llcrypt unregistered [ 9613.643433] Key type ._llcrypt registered [ 9613.645191] Key type .llcrypt registered [ 9613.687241] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9614.055436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9614.129619] alg: No test for adler32 (adler32-zlib) [ 9614.983193] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9615.070934] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9616.655126] Key type lgssc registered [ 9616.995797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9619.575825] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9619.581618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.039340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9640.159324] LustreError: 417803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 9666.783173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9668.702149] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9671.191995] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9673.534601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9673.733023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9673.760494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9675.125327] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9678.038403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9678.174116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.615979] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9682.154183] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9684.072854] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9686.497564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9686.501891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9686.503839] Lustre: Skipped 1 previous similar message [ 9691.616847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9691.619375] Lustre: Skipped 1 previous similar message [ 9692.629531] Lustre: server umount lustre-OST0000 complete [ 9696.737137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9696.738637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9696.740657] Lustre: Skipped 2 previous similar messages [ 9700.068663] Lustre: server umount lustre-MDT0000 complete [ 9701.328247] LustreError: 417811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838241 with bad export cookie 15318645456586261357 [ 9701.331162] LustreError: 417811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9701.331598] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9701.452672] Lustre: server umount lustre-MDT0001 complete [ 9705.172406] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9706.238275] Key type lgssc unregistered [ 9706.358403] LNet: 421372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9706.360702] LNetError: 421372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9706.369287] LNet: Removed LNI 192.168.202.159@tcp [ 9706.678304] Key type .llcrypt unregistered [ 9706.679409] Key type ._llcrypt unregistered [ 9716.185474] Key type ._llcrypt registered [ 9716.186941] Key type .llcrypt registered [ 9716.233195] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9716.725518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9716.740399] alg: No test for adler32 (adler32-zlib) [ 9717.611185] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9717.713538] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9719.303218] Key type lgssc registered [ 9719.705146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9722.403469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9722.413173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9723.594087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9724.925427] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9727.797553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.921302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9729.265282] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9731.760165] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9733.730439] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9735.973310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.094932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9738.064683] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9740.487306] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9748.395913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9753.569201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9753.570432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.577342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9753.579055] Lustre: Skipped 2 previous similar messages [ 9755.894670] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 9757.847897] Lustre: server umount lustre-MDT0000 complete [ 9758.603388] LustreError: 424603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.59@tcp arrived at 1786838298 with bad export cookie 7158562842036813910 [ 9758.607853] LustreError: 424603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9758.688969] LustreError: 422782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9758.694468] LustreError: 422782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9759.268553] LustreError: 422745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838299 with bad export cookie 7158562842036813287 [ 9759.272865] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9762.784106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9762.785975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.788643] Lustre: Skipped 2 previous similar messages [ 9762.790179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9763.808114] LustreError: 422763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.812390] LustreError: 422763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9764.832091] LustreError: 422782:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.832461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9764.837034] LustreError: 422782:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9764.838918] Lustre: Skipped 1 previous similar message [ 9765.393129] Lustre: server umount lustre-OST0000 complete [ 9767.763584] Lustre: server umount lustre-MDT0001 complete [ 9771.571455] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9772.560879] Key type lgssc unregistered [ 9772.701479] LNet: 426467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9772.704406] LNetError: 426467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9772.715361] LNet: Removed LNI 192.168.202.159@tcp [ 9773.038812] Key type .llcrypt unregistered [ 9773.040466] Key type ._llcrypt unregistered [ 9782.740772] Key type ._llcrypt registered [ 9782.741791] Key type .llcrypt registered [ 9782.785157] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9783.166180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9783.222110] alg: No test for adler32 (adler32-zlib) [ 9784.102880] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9784.214557] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9785.799136] Key type lgssc registered [ 9786.179428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9788.778482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9788.784368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9789.928694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9791.273783] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9794.091168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9794.222123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9795.591061] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9798.078944] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9800.019489] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9802.323647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9802.434915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9804.263768] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9806.645399] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9815.779459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9820.127936] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9820.129981] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.133825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9821.154616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9821.155056] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.453990] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 9822.456208] Lustre: Skipped 1 previous similar message [ 9824.410801] Lustre: server umount lustre-MDT0001 complete [ 9826.272631] LustreError: 427859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9826.273413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.273474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9826.278632] LustreError: 427859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9826.284369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9831.392383] LustreError: 427877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.393228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9832.279336] Lustre: server umount lustre-OST0000 complete [ 9833.708556] Lustre: server umount lustre-MDT0000 complete [ 9838.396878] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9839.545360] Key type lgssc unregistered [ 9839.686417] LNet: 431566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9839.688777] LNetError: 431566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9839.702295] LNet: Removed LNI 192.168.202.159@tcp [ 9840.037461] Key type .llcrypt unregistered [ 9840.039390] Key type ._llcrypt unregistered [ 9850.580730] Key type ._llcrypt registered [ 9850.582177] Key type .llcrypt registered [ 9850.626604] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 9858.704441] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9859.210847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9859.266056] alg: No test for adler32 (adler32-zlib) [ 9860.167828] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9860.308475] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9861.911163] Key type lgssc registered [ 9862.499864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9882.025417] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9888.987788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9888.993496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.077781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9890.089089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9890.128295] Lustre: lustre-MDT0000: new disk, initializing [ 9890.149179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9890.155796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9891.594837] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9897.323611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.426800] Lustre: lustre-OST0000: new disk, initializing [ 9897.428235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9897.429386] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9897.429748] Lustre: Skipped 1 previous similar message [ 9897.433614] Lustre: 436585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9897.454401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.356508] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9902.572239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9902.575534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9902.582660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9903.029979] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9908.787927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9908.833357] Lustre: 435563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9908.858837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9908.892270] Lustre: lustre-MDT0001: new disk, initializing [ 9908.932497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9908.947057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9908.950288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9910.377519] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 9914.377669] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9914.464311] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9916.143912] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9916.239168] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9918.432620] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9918.435535] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.436534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9918.441736] Lustre: Skipped 1 previous similar message [ 9921.274764] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 9921.279647] Lustre: Skipped 1 previous similar message [ 9923.553352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9923.555871] Lustre: Skipped 1 previous similar message [ 9924.310965] Lustre: server umount lustre-MDT0001 complete [ 9933.794108] LustreError: 435572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9933.801692] LustreError: 435572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9938.403260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9938.406863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9938.413304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9938.912493] LustreError: 435575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.033238] LustreError: 435576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.154317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9949.154713] LustreError: 435575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9949.160633] Lustre: Skipped 1 previous similar message [ 9951.199436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9951.260483] Lustre: server umount lustre-OST0000 complete [ 9958.908250] Lustre: server umount lustre-MDT0000 complete [ 9964.094865] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9965.275222] Key type lgssc unregistered [ 9965.413462] LNet: 440343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9965.416286] LNetError: 440343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9965.429555] LNet: Removed LNI 192.168.202.159@tcp [ 9965.811273] Key type .llcrypt unregistered [ 9965.812607] Key type ._llcrypt unregistered [ 9980.604039] Key type ._llcrypt registered [ 9980.605037] Key type .llcrypt registered [ 9980.652320] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 9980.788394] Key type .llcrypt unregistered [ 9980.790058] Key type ._llcrypt unregistered [ 9988.658705] Key type ._llcrypt registered [ 9988.659617] Key type .llcrypt registered [ 9988.705579] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 9989.183595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9989.200812] alg: No test for adler32 (adler32-zlib) [ 9990.069020] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 9990.167489] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 9991.767120] Key type lgssc registered [ 9992.214089] Lustre: Echo OBD driver; http://www.lustre.org/ [10003.505509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10003.511291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10004.603170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10004.616534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10004.664470] Lustre: lustre-MDT0000: new disk, initializing [10004.689468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10004.698157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10006.458874] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10011.187620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.233082] Lustre: 443695:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10011.259380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10011.262650] Lustre: Skipped 1 previous similar message [10011.307019] Lustre: lustre-MDT0001: new disk, initializing [10011.348629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10011.367646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10011.371600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10012.986664] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10017.400244] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10019.879121] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10022.911655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10023.029722] Lustre: lustre-OST0000: new disk, initializing [10023.031434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10023.034431] Lustre: 445662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10023.066688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10025.280127] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10029.960210] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10030.599355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10030.603365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10030.648196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10040.802247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.803307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10040.808852] Lustre: Skipped 1 previous similar message [10044.322049] Lustre: server umount lustre-MDT0000 complete [10045.974370] LustreError: 443686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838586 with bad export cookie 10687302629510658276 [10045.977619] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10045.979386] LustreError: 443686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10046.115524] Lustre: server umount lustre-MDT0001 complete [10057.947941] Lustre: server umount lustre-OST0000 complete [10062.174791] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:03:22 (1786838602) [10065.452511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.645413] LustreError: 447686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.653373] LustreError: 447686:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10065.681301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10067.355064] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10071.010739] LustreError: 447687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.075399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.232702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10073.116079] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10076.320035] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10078.616263] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10094.626325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10094.733165] Lustre: Found index 0 for lustre-OST0000, updating log [10094.820416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10097.571316] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10100.700490] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10111.205203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10116.580883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.581435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10116.586057] Lustre: Skipped 1 previous similar message [10116.586946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.593735] Lustre: Skipped 3 previous similar messages [10117.601920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10117.604429] Lustre: Skipped 1 previous similar message [10118.823857] Lustre: server umount lustre-OST0000 complete [10131.796498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.802068] Lustre: Skipped 1 previous similar message [10131.806053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10131.808779] Lustre: Skipped 1 previous similar message [10137.056515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10137.060199] Lustre: Skipped 1 previous similar message [10146.271137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10146.344800] Lustre: server umount lustre-MDT0000 complete [10148.067741] LustreError: 447666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786838688 with bad export cookie 10687302629510659655 [10148.071459] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10148.072568] LustreError: 447666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10148.221617] Lustre: server umount lustre-MDT0001 complete [10152.654818] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:04:52 (1786838692) [10163.232723] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp [10734.794016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.949432] LustreError: 454113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.982746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10736.391993] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10739.439242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10739.571267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10740.969089] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10743.480785] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10745.356364] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10760.802872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10760.915248] Lustre: Found index 0 for lustre-OST0000, updating log [10761.035177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10763.052251] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10763.110482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [10765.565738] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10767.332362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10767.332629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.335168] Lustre: Skipped 6 previous similar messages [10767.339991] Lustre: Skipped 1 previous similar message [10772.364865] Lustre: server umount lustre-OST0000 complete [10774.624407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10774.625640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10774.628036] Lustre: Skipped 1 previous similar message [10776.672136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10776.674571] Lustre: Skipped 4 previous similar messages [10779.734329] Lustre: server umount lustre-MDT0000 complete [10779.745277] LustreError: 454113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.751811] LustreError: 454113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10780.768684] LustreError: 454853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.774375] LustreError: 454853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10781.074486] LustreError: 454093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839321 with bad export cookie 10687302629510660726 [10781.077440] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10781.078919] LustreError: 454093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10781.203375] Lustre: server umount lustre-MDT0001 complete [10791.968426] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp [10798.982631] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [10802.612616] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10815.997388] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:15:56 (1786839356) [10822.096388] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10823.188687] Key type lgssc unregistered [10823.321446] LNet: 460764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10823.324363] LNetError: 460764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10892.966148] LNet: Removed LNI 192.168.202.159@tcp [10893.289657] Key type .llcrypt unregistered [10893.290821] Key type ._llcrypt unregistered [10900.266714] Key type ._llcrypt registered [10900.267950] Key type .llcrypt registered [10900.313738] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10900.684913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10900.731814] alg: No test for adler32 (adler32-zlib) [10901.589678] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [10901.686396] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [10903.271179] Key type lgssc registered [10903.645240] Lustre: Echo OBD driver; http://www.lustre.org/ [10905.146797] LNet: Added LNI 192.168.202.159@tcp100 [8/256/0/180] [10905.466804] LNet: Added LNI 192.168.202.159@tcp101 [8/256/0/180] [10905.790337] LNet: Added LNI 192.168.202.159@tcp102 [8/256/0/180] [10906.097113] LNet: Added LNI 192.168.202.159@tcp103 [8/256/0/180] [10906.435712] LNet: Added LNI 192.168.202.159@tcp104 [8/256/0/180] [10906.819225] LNet: Added LNI 192.168.202.159@tcp105 [8/256/0/180] [10907.163083] LNet: Added LNI 192.168.202.159@tcp106 [8/256/0/180] [10907.517525] LNet: Added LNI 192.168.202.159@tcp107 [8/256/0/180] [10907.888207] LNet: Added LNI 192.168.202.159@tcp108 [8/256/0/180] [10908.224767] LNet: Added LNI 192.168.202.159@tcp109 [8/256/0/180] [10908.556588] LNet: Added LNI 192.168.202.159@tcp110 [8/256/0/180] [10908.893337] LNet: Added LNI 192.168.202.159@tcp111 [8/256/0/180] [10909.218078] LNet: Added LNI 192.168.202.159@tcp112 [8/256/0/180] [10909.532750] LNet: Added LNI 192.168.202.159@tcp113 [8/256/0/180] [10909.846622] LNet: Added LNI 192.168.202.159@tcp114 [8/256/0/180] [10910.171977] LNet: Added LNI 192.168.202.159@tcp115 [8/256/0/180] [10910.493370] LNet: Added LNI 192.168.202.159@tcp116 [8/256/0/180] [10910.821715] LNet: Added LNI 192.168.202.159@tcp117 [8/256/0/180] [10911.152142] LNet: Added LNI 192.168.202.159@tcp118 [8/256/0/180] [10911.481650] LNet: Added LNI 192.168.202.159@tcp119 [8/256/0/180] [10911.792376] LNet: Added LNI 192.168.202.159@tcp120 [8/256/0/180] [10912.126270] LNet: Added LNI 192.168.202.159@tcp121 [8/256/0/180] [10912.470771] LNet: Added LNI 192.168.202.159@tcp122 [8/256/0/180] [10912.828377] LNet: Added LNI 192.168.202.159@tcp123 [8/256/0/180] [10913.175408] LNet: Added LNI 192.168.202.159@tcp124 [8/256/0/180] [10913.491919] LNet: Added LNI 192.168.202.159@tcp125 [8/256/0/180] [10913.808108] LNet: Added LNI 192.168.202.159@tcp126 [8/256/0/180] [10914.172684] LNet: Added LNI 192.168.202.159@tcp127 [8/256/0/180] [10914.533516] LNet: Added LNI 192.168.202.159@tcp128 [8/256/0/180] [10914.849613] LNet: Added LNI 192.168.202.159@tcp129 [8/256/0/180] [10915.157884] LNet: Added LNI 192.168.202.159@tcp130 [8/256/0/180] [10915.463181] LNet: Added LNI 192.168.202.159@tcp131 [8/256/0/180] [10915.774511] LNet: Added LNI 192.168.202.159@tcp132 [8/256/0/180] [10916.079517] LNet: Added LNI 192.168.202.159@tcp133 [8/256/0/180] [10916.390891] LNet: Added LNI 192.168.202.159@tcp134 [8/256/0/180] [10916.711269] LNet: Added LNI 192.168.202.159@tcp135 [8/256/0/180] [10919.092755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10919.098088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.134968] Lustre: 463760:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [10920.160593] LNetError: 461226:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.159 [10920.180174] LNetError: Protocol error connecting to 192.168.202.159@tcp120 at host 192.168.202.159:988: is it running a compatible version of Lustre? [10920.183901] LNetError: Skipped 1 previous similar message [10920.226650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10920.237088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10920.273179] Lustre: lustre-MDT0000: new disk, initializing [10920.294738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10920.301271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10921.184760] LNet: 1 local NIs in recovery (showing 1): 192.168.202.159@tcp120 [10921.185716] LNetError: 463786:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.159 [10921.190643] LNetError: 463786:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [10921.195594] LNetError: Protocol error connecting to 192.168.202.159@tcp104 at host 192.168.202.159:988: is it running a compatible version of Lustre? [10921.199413] LNetError: Skipped 3 previous similar messages [10921.711294] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10923.232237] LNetError: 461224:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.159 [10923.235048] LNetError: 461224:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [10923.237741] LNetError: Protocol error connecting to 192.168.202.159@tcp120 at host 192.168.202.159:988: is it running a compatible version of Lustre? [10926.016443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.081125] Lustre: 463775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10926.110248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10926.113305] Lustre: Skipped 1 previous similar message [10926.156545] Lustre: lustre-MDT0001: new disk, initializing [10926.169836] Lustre: 464581:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [10926.173663] Lustre: 464581:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10926.228028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10926.241406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10926.244871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10927.329164] LNetError: 463785:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.159 [10927.333627] LNetError: Protocol error connecting to 192.168.202.159@tcp120 at host 192.168.202.159:988: is it running a compatible version of Lustre? [10927.752060] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10931.614334] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10933.503270] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10935.520453] LNetError: 461225:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.159 [10935.526501] LNetError: Protocol error connecting to 192.168.202.159@tcp120 at host 192.168.202.159:988: is it running a compatible version of Lustre? [10935.799240] Lustre: 465692:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [10935.802558] Lustre: 465692:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10936.081220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10936.204496] Lustre: lustre-OST0000: new disk, initializing [10936.206352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10936.209574] Lustre: 465753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10936.264470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10938.348646] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [10942.114523] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10944.786904] Lustre: server umount lustre-OST0000 complete [10945.014532] Lustre: 466719:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.159@tcp: can't add NID 192.168.202.159@tcp130: rc = -75 [10945.018858] Lustre: 466719:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [10947.039825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10947.043279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10947.048980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10950.112435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10950.112796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10950.815978] LNetError: 463782:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.159 [10950.819221] LNetError: Protocol error connecting to 192.168.202.159@tcp126 at host 192.168.202.159:988: is it running a compatible version of Lustre? [10952.287754] Lustre: server umount lustre-MDT0000 complete [10953.566277] LustreError: 463768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839493 with bad export cookie 2038753720541234177 [10953.569283] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10953.570038] LustreError: 463768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10953.682312] Lustre: server umount lustre-MDT0001 complete [10957.420494] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [10958.539969] Key type lgssc unregistered [10958.707282] LNet: 467663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10958.711350] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10958.714096] LNet: Removed LNI 192.168.202.159@tcp [10959.775138] LNet: Removed LNI 192.168.202.159@tcp100 [10959.776643] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10959.779251] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10959.781774] LNet: Removed LNI 192.168.202.159@tcp101 [10960.799144] LNet: Removed LNI 192.168.202.159@tcp102 [10960.800684] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10960.802979] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10960.805495] LNet: Removed LNI 192.168.202.159@tcp103 [10961.823185] LNet: Removed LNI 192.168.202.159@tcp104 [10962.847136] LNet: Removed LNI 192.168.202.159@tcp105 [10962.848489] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10962.850956] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10962.853505] LNet: Removed LNI 192.168.202.159@tcp106 [10963.871167] LNet: Removed LNI 192.168.202.159@tcp107 [10964.895158] LNet: Removed LNI 192.168.202.159@tcp108 [10965.919140] LNet: Removed LNI 192.168.202.159@tcp109 [10966.943123] LNet: Removed LNI 192.168.202.159@tcp110 [10966.944433] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10966.946894] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10966.949434] LNet: Removed LNI 192.168.202.159@tcp111 [10967.967141] LNet: Removed LNI 192.168.202.159@tcp112 [10968.991100] LNet: Removed LNI 192.168.202.159@tcp113 [10970.015160] LNet: Removed LNI 192.168.202.159@tcp114 [10971.039144] LNet: Removed LNI 192.168.202.159@tcp115 [10972.063110] LNet: Removed LNI 192.168.202.159@tcp116 [10973.087164] LNet: Removed LNI 192.168.202.159@tcp117 [10974.111145] LNet: Removed LNI 192.168.202.159@tcp118 [10975.135130] LNet: Removed LNI 192.168.202.159@tcp119 [10975.136637] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10975.138837] LNetError: 467663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10975.141466] LNet: Removed LNI 192.168.202.159@tcp120 [10976.159150] LNet: Removed LNI 192.168.202.159@tcp121 [10977.183080] LNet: Removed LNI 192.168.202.159@tcp122 [10978.207097] LNet: Removed LNI 192.168.202.159@tcp123 [10979.231124] LNet: Removed LNI 192.168.202.159@tcp124 [10980.255181] LNet: Removed LNI 192.168.202.159@tcp125 [10981.279212] LNet: Removed LNI 192.168.202.159@tcp126 [10982.303170] LNet: Removed LNI 192.168.202.159@tcp127 [10983.327481] LNet: Removed LNI 192.168.202.159@tcp128 [10983.329169] LNet: Removed LNI 192.168.202.159@tcp129 [10983.330637] LNet: Removed LNI 192.168.202.159@tcp130 [10983.332064] LNet: Removed LNI 192.168.202.159@tcp131 [10983.333740] LNet: Removed LNI 192.168.202.159@tcp132 [10983.335110] LNet: Removed LNI 192.168.202.159@tcp133 [10983.336333] LNet: Removed LNI 192.168.202.159@tcp134 [10983.354142] LNet: Removed LNI 192.168.202.159@tcp135 [10983.682605] Key type .llcrypt unregistered [10983.684326] Key type ._llcrypt unregistered [10991.377528] Key type ._llcrypt registered [10991.378510] Key type .llcrypt registered [10991.412176] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [10997.162852] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [10997.572932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10997.603777] alg: No test for adler32 (adler32-zlib) [10998.450111] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [10998.536607] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11000.111175] Key type lgssc registered [11000.473185] Lustre: Echo OBD driver; http://www.lustre.org/ [11011.183251] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:19:11 (1786839551) [11016.979355] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11017.992492] Key type lgssc unregistered [11018.115299] LNet: 471726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11018.118052] LNetError: 471726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11018.127286] LNet: Removed LNI 192.168.202.159@tcp [11018.414497] Key type .llcrypt unregistered [11018.416283] Key type ._llcrypt unregistered [11025.034582] Key type ._llcrypt registered [11025.035844] Key type .llcrypt registered [11025.069353] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11025.419946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11025.450096] alg: No test for adler32 (adler32-zlib) [11026.304279] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [11026.390825] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11027.975146] Key type lgssc registered [11028.301746] Lustre: Echo OBD driver; http://www.lustre.org/ [11031.328811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11031.334226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11032.417150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11032.427631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11032.465715] Lustre: lustre-MDT0000: new disk, initializing [11032.486053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11032.492329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11033.848445] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11037.986794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11038.034981] Lustre: 472915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11038.068722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11038.071344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11038.073162] Lustre: Skipped 1 previous similar message [11038.113931] Lustre: lustre-MDT0001: new disk, initializing [11038.149588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11038.162807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11038.165827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11039.473690] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11043.110771] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11044.931694] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11047.218498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11047.339387] Lustre: lustre-OST0000: new disk, initializing [11047.341602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11047.344566] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [11047.344920] Lustre: 474884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11047.384427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11049.315516] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11052.525867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11052.530186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11052.542577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11052.961580] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11054.085071] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing lnet_if_list [11055.640496] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11055.649714] LNet: Added LNI 192.168.202.159@tcp10 [8/256/0/180] [11057.420604] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11057.431834] LNet: Added LNI 192.168.202.159@tcp11 [8/256/0/180] [11058.623476] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11058.632519] LNet: Added LNI 192.168.202.159@tcp12 [8/256/0/180] [11059.864331] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [11059.874471] LNet: Added LNI 192.168.202.159@tcp13 [8/256/0/180] [11061.205366] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [11061.216832] LNet: Added LNI 192.168.202.159@tcp14 [8/256/0/180] [11062.495375] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [11062.506544] LNet: Added LNI 192.168.202.159@tcp15 [8/256/0/180] [11063.883494] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [11063.893454] LNet: Added LNI 192.168.202.159@tcp16 [8/256/0/180] [11065.274723] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [11065.285698] LNet: Added LNI 192.168.202.159@tcp17 [8/256/0/180] [11066.677841] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11066.688225] LNet: Added LNI 192.168.202.159@tcp18 [8/256/0/180] [11068.232832] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11068.245937] LNet: Added LNI 192.168.202.159@tcp19 [8/256/0/180] [11069.817633] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11069.827473] LNet: Added LNI 192.168.202.159@tcp20 [8/256/0/180] [11071.402351] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11071.413559] LNet: Added LNI 192.168.202.159@tcp21 [8/256/0/180] [11072.938190] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11072.948876] LNet: Added LNI 192.168.202.159@tcp22 [8/256/0/180] [11074.514100] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11074.525407] LNet: Added LNI 192.168.202.159@tcp23 [8/256/0/180] [11076.142532] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11076.152521] LNet: Added LNI 192.168.202.159@tcp24 [8/256/0/180] [11077.774277] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11077.784772] LNet: Added LNI 192.168.202.159@tcp25 [8/256/0/180] [11079.532466] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11079.543796] LNet: Added LNI 192.168.202.159@tcp26 [8/256/0/180] [11081.294269] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11081.306899] LNet: Added LNI 192.168.202.159@tcp27 [8/256/0/180] [11083.109922] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11083.120433] LNet: Added LNI 192.168.202.159@tcp28 [8/256/0/180] [11084.907263] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11084.919149] LNet: Added LNI 192.168.202.159@tcp29 [8/256/0/180] [11086.772706] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11086.783725] LNet: Added LNI 192.168.202.159@tcp30 [8/256/0/180] [11088.762587] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11088.772873] LNet: Added LNI 192.168.202.159@tcp31 [8/256/0/180] [11090.772647] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11090.783268] LNet: Added LNI 192.168.202.159@tcp32 [8/256/0/180] [11092.841084] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11092.853216] LNet: Added LNI 192.168.202.159@tcp33 [8/256/0/180] [11095.132118] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11095.145273] LNet: Added LNI 192.168.202.159@tcp34 [8/256/0/180] [11097.410647] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11097.424168] LNet: Added LNI 192.168.202.159@tcp35 [8/256/0/180] [11099.730642] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11099.744416] LNet: Added LNI 192.168.202.159@tcp36 [8/256/0/180] [11101.919528] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11101.932426] LNet: Added LNI 192.168.202.159@tcp37 [8/256/0/180] [11104.102291] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11104.113684] LNet: Added LNI 192.168.202.159@tcp38 [8/256/0/180] [11106.193220] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11106.205401] LNet: Added LNI 192.168.202.159@tcp39 [8/256/0/180] [11108.401321] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11108.411766] LNet: Added LNI 192.168.202.159@tcp40 [8/256/0/180] [11110.743597] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11110.755445] LNet: Added LNI 192.168.202.159@tcp41 [8/256/0/180] [11113.192736] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11113.206535] LNet: Added LNI 192.168.202.159@tcp42 [8/256/0/180] [11115.619121] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11115.631480] LNet: Added LNI 192.168.202.159@tcp43 [8/256/0/180] [11118.143684] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11118.155878] LNet: Added LNI 192.168.202.159@tcp44 [8/256/0/180] [11120.601185] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11120.611951] LNet: Added LNI 192.168.202.159@tcp45 [8/256/0/180] [11123.039185] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11123.052630] LNet: Added LNI 192.168.202.159@tcp46 [8/256/0/180] [11125.519593] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11125.528793] LNet: Added LNI 192.168.202.159@tcp47 [8/256/0/180] [11128.024624] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11128.037070] LNet: Added LNI 192.168.202.159@tcp48 [8/256/0/180] [11130.688936] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11130.700872] LNet: Added LNI 192.168.202.159@tcp49 [8/256/0/180] [11133.264796] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11133.275437] LNet: Added LNI 192.168.202.159@tcp50 [8/256/0/180] [11144.676381] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11144.678020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.679879] Lustre: Skipped 2 previous similar messages [11144.683258] Lustre: Skipped 2 previous similar messages [11146.776623] Lustre: server umount lustre-MDT0000 complete [11147.952194] LustreError: 472907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839688 with bad export cookie 3051265490830260664 [11147.954922] LustreError: 472907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11147.955435] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11148.069570] Lustre: server umount lustre-MDT0001 complete [11154.871606] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11157.619905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11157.919469] LustreError: 472187:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000456fb013 x1873637084840064/t0(0) o250->MGC192.168.202.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11158.015486] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11158.028835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11159.337512] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11162.105553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11162.222181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11163.551812] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11164.259812] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11165.279255] Lustre: 472191:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786839689/real 1786839689] req@00000000163e5163 x1873637084839680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786839705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11165.827340] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11167.596806] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11168.207183] Lustre: 472188:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786839692/real 1786839692] req@00000000d7bee3e2 x1873637084839808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786839708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11170.783137] Lustre: 472189:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786839694/real 1786839694] req@00000000d9cb8101 x1873637084839936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786839710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11174.431084] Lustre: 472190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786839698/real 1786839698] req@00000000e37207ae x1873637084840192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786839714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11174.881692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11174.882969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11174.886528] Lustre: Skipped 1 previous similar message [11180.003107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11180.004881] Lustre: Skipped 2 previous similar messages [11180.566333] Lustre: server umount lustre-OST0000 complete [11182.559731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11182.562420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11190.240072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11190.242065] Lustre: Skipped 4 previous similar messages [11196.383117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11196.440758] Lustre: server umount lustre-MDT0000 complete [11197.604041] LustreError: 472906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839737 with bad export cookie 3051265490830262722 [11197.607134] LustreError: 472906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11197.607142] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11197.711558] Lustre: server umount lustre-MDT0001 complete [11201.054686] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11202.061616] Key type lgssc unregistered [11202.219483] LNet: 501843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11202.223444] LNetError: 501843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11202.225839] LNet: Removed LNI 192.168.202.159@tcp [11202.227450] LNet: Removed LNI 192.168.202.159@tcp10 [11202.229124] LNet: Removed LNI 192.168.202.159@tcp11 [11202.230541] LNet: Removed LNI 192.168.202.159@tcp12 [11202.232219] LNet: Removed LNI 192.168.202.159@tcp13 [11202.233607] LNet: Removed LNI 192.168.202.159@tcp14 [11202.235159] LNet: Removed LNI 192.168.202.159@tcp15 [11202.236536] LNet: Removed LNI 192.168.202.159@tcp16 [11202.238186] LNet: Removed LNI 192.168.202.159@tcp17 [11202.239406] LNet: Removed LNI 192.168.202.159@tcp18 [11202.240988] LNet: Removed LNI 192.168.202.159@tcp19 [11202.242694] LNet: Removed LNI 192.168.202.159@tcp20 [11202.244032] LNet: Removed LNI 192.168.202.159@tcp21 [11202.245094] LNet: Removed LNI 192.168.202.159@tcp22 [11202.246115] LNet: Removed LNI 192.168.202.159@tcp23 [11202.247222] LNet: Removed LNI 192.168.202.159@tcp24 [11202.248348] LNet: Removed LNI 192.168.202.159@tcp25 [11202.249524] LNet: Removed LNI 192.168.202.159@tcp26 [11202.250683] LNet: Removed LNI 192.168.202.159@tcp27 [11202.251888] LNet: Removed LNI 192.168.202.159@tcp28 [11202.253077] LNet: Removed LNI 192.168.202.159@tcp29 [11202.254171] LNet: Removed LNI 192.168.202.159@tcp30 [11202.255427] LNet: Removed LNI 192.168.202.159@tcp31 [11202.256563] LNet: Removed LNI 192.168.202.159@tcp32 [11202.257684] LNet: Removed LNI 192.168.202.159@tcp33 [11202.258882] LNet: Removed LNI 192.168.202.159@tcp34 [11202.260315] LNet: Removed LNI 192.168.202.159@tcp35 [11202.262169] LNet: Removed LNI 192.168.202.159@tcp36 [11202.263962] LNet: Removed LNI 192.168.202.159@tcp37 [11202.265846] LNet: Removed LNI 192.168.202.159@tcp38 [11202.267031] LNet: Removed LNI 192.168.202.159@tcp39 [11202.268280] LNet: Removed LNI 192.168.202.159@tcp40 [11202.269351] LNet: Removed LNI 192.168.202.159@tcp41 [11202.270666] LNet: Removed LNI 192.168.202.159@tcp42 [11202.272639] LNet: Removed LNI 192.168.202.159@tcp43 [11202.273985] LNet: Removed LNI 192.168.202.159@tcp44 [11202.275069] LNet: Removed LNI 192.168.202.159@tcp45 [11202.276176] LNet: Removed LNI 192.168.202.159@tcp46 [11202.277220] LNet: Removed LNI 192.168.202.159@tcp47 [11202.278257] LNet: Removed LNI 192.168.202.159@tcp48 [11202.279320] LNet: Removed LNI 192.168.202.159@tcp49 [11202.284271] LNet: Removed LNI 192.168.202.159@tcp50 [11202.557076] Key type .llcrypt unregistered [11202.558459] Key type ._llcrypt unregistered [11209.191687] Key type ._llcrypt registered [11209.192646] Key type .llcrypt registered [11209.231834] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [11215.729710] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11216.172158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11216.181934] alg: No test for adler32 (adler32-zlib) [11217.058123] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [11217.148602] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11218.735193] Key type lgssc registered [11219.105279] Lustre: Echo OBD driver; http://www.lustre.org/ [11230.969926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11230.976350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11232.059328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11232.071376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11232.108555] Lustre: lustre-MDT0000: new disk, initializing [11232.129416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11232.135624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11233.488964] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11237.747908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11237.788073] Lustre: 505012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11237.810955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11237.812967] Lustre: Skipped 1 previous similar message [11237.852892] Lustre: lustre-MDT0001: new disk, initializing [11237.894767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11237.911929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11237.915577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11239.219914] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11242.854639] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11244.648290] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11247.255295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.329203] Lustre: *** cfs_fail_loc=90f, val=0*** [11247.363103] Lustre: lustre-OST0000: new disk, initializing [11247.365167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11247.368128] Lustre: 507026:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11247.397865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11249.328790] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11250.486049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11250.489668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11250.520537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11252.980354] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11255.777799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11255.778062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11255.778597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11255.780854] LustreError: Skipped 1 previous similar message [11255.784378] Lustre: Skipped 1 previous similar message [11260.897216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11260.899569] Lustre: Skipped 2 previous similar messages [11261.388439] Lustre: server umount lustre-OST0000 complete [11263.455986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11263.458609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.463143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11266.015941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11266.016568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11268.759483] Lustre: server umount lustre-MDT0000 complete [11269.934996] LustreError: 505049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839810 with bad export cookie 15158264547525811461 [11269.937860] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11269.938920] LustreError: 505049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11270.052197] Lustre: server umount lustre-MDT0001 complete [11273.517966] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11274.546218] Key type lgssc unregistered [11274.671346] LNet: 508932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11274.673399] LNetError: 508932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11274.680289] LNet: Removed LNI 192.168.202.159@tcp [11274.975524] Key type .llcrypt unregistered [11274.976681] Key type ._llcrypt unregistered [11282.138558] Key type ._llcrypt registered [11282.139978] Key type .llcrypt registered [11282.174908] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11282.475167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11282.522137] alg: No test for adler32 (adler32-zlib) [11283.376846] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [11283.457358] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11285.039165] Key type lgssc registered [11285.378846] Lustre: Echo OBD driver; http://www.lustre.org/ [11287.706081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11287.711607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11288.843485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11290.122575] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11292.922565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.060900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11294.450486] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11296.841897] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11298.631316] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11300.714293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11300.821599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11302.613628] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11304.963276] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11319.072516] loop0: detected capacity change from 0 to 204800000 [11319.131368] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11319.135382] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11319.141690] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11321.191044] loop0: detected capacity change from 0 to 204800000 [11321.310932] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11321.313948] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11321.316937] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11322.991827] loop0: detected capacity change from 0 to 204800000 [11324.887391] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.936747] Lustre: fs15246-OST0000: new disk, initializing [11324.938283] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11324.940453] Lustre: 513794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11324.959061] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11326.834562] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11330.562888] loop1: detected capacity change from 0 to 204800000 [11332.411319] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11332.450830] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11332.470139] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11332.505499] Lustre: fs15246-MDT0000: new disk, initializing [11332.555982] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11332.565101] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11332.603159] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11333.664039] Lustre: 514915:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d7bee3e2 x1873637354273408/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786839889 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11334.011686] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11341.286161] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11342.751248] LustreError: 514915:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [11342.755174] LustreError: 514915:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [11342.758933] LustreError: 514915:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [11346.402555] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11347.081846] Lustre: server umount fs15246-MDT0000 complete [11349.953133] Lustre: server umount fs15246-OST0000 complete [11354.079767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11354.082418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11354.085985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11355.103812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11355.106182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.641050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11356.643488] Lustre: Skipped 2 previous similar messages [11361.760876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11361.763114] Lustre: Skipped 1 previous similar message [11367.391153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11367.445817] Lustre: server umount lustre-OST0000 complete [11369.951845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11369.954578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.958983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11369.961064] Lustre: Skipped 2 previous similar messages [11372.001062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11374.868189] Lustre: server umount lustre-MDT0000 complete [11376.213258] LustreError: 510159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839916 with bad export cookie 18422921031317316870 [11376.216904] LustreError: 510159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11376.217116] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11376.340100] Lustre: server umount lustre-MDT0001 complete [11379.877278] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11380.914114] Key type lgssc unregistered [11381.037347] LNet: 518009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11381.039272] LNetError: 518009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11381.047262] LNet: Removed LNI 192.168.202.159@tcp [11381.330989] Key type .llcrypt unregistered [11381.332447] Key type ._llcrypt unregistered [11388.682084] Key type ._llcrypt registered [11388.683639] Key type .llcrypt registered [11388.722660] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11389.194885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11389.205347] alg: No test for adler32 (adler32-zlib) [11390.064503] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [11390.157170] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [11391.735164] Key type lgssc registered [11392.110981] Lustre: Echo OBD driver; http://www.lustre.org/ [11404.446715] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [11407.821806] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11419.817202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11419.822190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11420.909221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11420.920811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11420.960573] Lustre: lustre-MDT0000: new disk, initializing [11420.985132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11420.991617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11422.380035] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11426.488170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11426.532582] Lustre: 522442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11426.554946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11426.556713] Lustre: Skipped 1 previous similar message [11426.594820] Lustre: lustre-MDT0001: new disk, initializing [11426.624606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11426.638670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11426.642912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11427.943569] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11431.609976] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11433.454729] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11435.776061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11435.879431] Lustre: lustre-OST0000: new disk, initializing [11435.882200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11435.885771] Lustre: 524409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11435.915887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11437.850122] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [11441.145317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11441.150428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11441.184170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11441.648382] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11443.092971] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11443.163762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11444.599508] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11444.675900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11446.241912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11446.242124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11446.243334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11446.246131] Lustre: Skipped 1 previous similar message [11446.251298] LustreError: Skipped 1 previous similar message [11447.440285] Lustre: server umount lustre-OST0000 complete [11451.360820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11451.361521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11451.365416] Lustre: Skipped 1 previous similar message [11451.369378] Lustre: Skipped 2 previous similar messages [11454.812327] Lustre: server umount lustre-MDT0000 complete [11456.068308] LustreError: 523237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786839996 with bad export cookie 10532033301045698755 [11456.071019] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11456.071821] LustreError: 523237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11456.190143] Lustre: server umount lustre-MDT0001 complete [11463.998145] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [11467.581818] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [11478.325860] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11242 sec ========= 20:26:59 (1786840019) [11478.883175] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:26:59 (1786840019) === [11480.061597] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:27:00 (1786840020) === [11493.750687] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [11494.786948] Key type lgssc unregistered [11494.926348] LNet: 531043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11494.928679] LNetError: 531043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11494.936335] LNet: Removed LNI 192.168.202.159@tcp [11495.270553] Key type .llcrypt unregistered [11495.271906] Key type ._llcrypt unregistered