[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 480440637 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002365] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006870] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.020831] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025285] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027090] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] *** VALIDATE tmpfs *** [ 0.030497] *** VALIDATE proc *** [ 0.032009] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040898] debug: unmapping init [mem 0xffffffffb6859000-0xffffffffb6860fff] [ 0.042850] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050013] ... event mask: 000000070000000f [ 0.051265] rcu: Hierarchical SRCU implementation. [ 0.053516] smp: Bringing up secondary CPUs ... [ 0.054623] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.059417] smp: Brought up 1 node, 4 CPUs [ 0.061018] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.252858] node 0 deferred pages initialised in 187ms [ 0.256359] devtmpfs: initialized [ 0.257175] x86/mm: Memory block size: 128MB [ 0.259210] gcov: version magic: 0x41383552 [ 0.261380] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.264165] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.267400] pinctrl core: initialized pinctrl subsystem [ 0.269145] [ 0.269532] ************************************************************* [ 0.271011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272009] ** ** [ 0.274016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.276016] ** ** [ 0.278013] ** This means that this kernel is built to expose internal ** [ 0.280017] ** IOMMU data structures, which may compromise security on ** [ 0.282017] ** your system. ** [ 0.284015] ** ** [ 0.287016] ** If you see this message and you are not debugging the ** [ 0.288012] ** kernel, report this immediately to your vendor! ** [ 0.290016] ** ** [ 0.293019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.295013] ************************************************************* [ 0.297826] NET: Registered protocol family 16 [ 0.299459] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.302084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.308597] cpuidle: using governor menu [ 0.310811] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.312609] PCI: Using configuration type 1 for base access [ 0.315121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.325084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.328043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.329167] cryptd: max_cpu_qlen set to 1000 [ 0.331189] ACPI: Added _OSI(Module Device) [ 0.332030] ACPI: Added _OSI(Processor Device) [ 0.333017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.335013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.339000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.345702] ACPI: Interpreter enabled [ 0.347087] ACPI: PM: (supports S0 S3 S4 S5) [ 0.348025] ACPI: Using IOAPIC for interrupt routing [ 0.350187] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354462] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.366394] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.368057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.370028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.374114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.378627] acpiphp: Slot [2] registered [ 0.380148] acpiphp: Slot [5] registered [ 0.382161] acpiphp: Slot [6] registered [ 0.383126] acpiphp: Slot [7] registered [ 0.384138] acpiphp: Slot [8] registered [ 0.385116] acpiphp: Slot [9] registered [ 0.386038] acpiphp: Slot [10] registered [ 0.386955] acpiphp: Slot [3] registered [ 0.388095] acpiphp: Slot [4] registered [ 0.389128] acpiphp: Slot [11] registered [ 0.391106] acpiphp: Slot [12] registered [ 0.392125] acpiphp: Slot [13] registered [ 0.393134] acpiphp: Slot [14] registered [ 0.395122] acpiphp: Slot [15] registered [ 0.396121] acpiphp: Slot [16] registered [ 0.397124] acpiphp: Slot [17] registered [ 0.398101] acpiphp: Slot [18] registered [ 0.399113] acpiphp: Slot [19] registered [ 0.400007] acpiphp: Slot [20] registered [ 0.400902] acpiphp: Slot [21] registered [ 0.402106] acpiphp: Slot [22] registered [ 0.403095] acpiphp: Slot [23] registered [ 0.403995] acpiphp: Slot [24] registered [ 0.405063] acpiphp: Slot [25] registered [ 0.405936] acpiphp: Slot [26] registered [ 0.406068] acpiphp: Slot [27] registered [ 0.406932] acpiphp: Slot [28] registered [ 0.408104] acpiphp: Slot [29] registered [ 0.408911] acpiphp: Slot [30] registered [ 0.410065] acpiphp: Slot [31] registered [ 0.410868] PCI host bridge to bus 0000:00 [ 0.411021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.413029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.415018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.417031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.419030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.422038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.424218] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.427083] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.429365] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.437023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.442807] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.445026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.447022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.451612] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.454014] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.456050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.458633] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.474845] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.479011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.484830] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.490022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.496024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525106] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.535022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.549022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.555000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.560027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.569024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.580024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.588685] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.598020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.603018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.617025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625556] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.639020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.652047] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.668024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.695025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.707428] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.710692] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712317] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.713147] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.716158] iommu: Default domain type: Passthrough [ 0.717396] SCSI subsystem initialized [ 0.718117] ACPI: bus type USB registered [ 0.720088] usbcore: registered new interface driver usbfs [ 0.722110] usbcore: registered new interface driver hub [ 0.723093] usbcore: registered new device driver usb [ 0.724166] pps_core: LinuxPPS API ver. 1 registered [ 0.726013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729087] PTP clock support registered [ 0.731133] EDAC MC: Ver: 3.0.0 [ 0.733203] PCI: Using ACPI for IRQ routing [ 0.734972] NetLabel: Initializing [ 0.736015] NetLabel: domain hash size = 128 [ 0.737008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738112] NetLabel: unlabeled traffic allowed by default [ 0.739139] vgaarb: loaded [ 0.740362] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750006] clocksource: Switched to clocksource kvm-clock [ 0.845949] VFS: Disk quotas dquot_6.6.0 [ 0.847579] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.849488] *** VALIDATE ramfs *** [ 0.850591] *** VALIDATE hugetlbfs *** [ 0.851819] pnp: PnP ACPI init [ 0.854654] pnp: PnP ACPI: found 6 devices [ 0.869455] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.872922] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.875190] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.877390] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.879711] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.881960] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.884733] NET: Registered protocol family 2 [ 0.886754] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.890906] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.895444] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.900824] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.904242] TCP: Hash tables configured (established 65536 bind 65536) [ 0.907377] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.910541] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913563] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.916372] NET: Registered protocol family 1 [ 0.918722] RPC: Registered named UNIX socket transport module. [ 0.920186] RPC: Registered udp transport module. [ 0.921087] RPC: Registered tcp transport module. [ 0.922368] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.923926] NET: Registered protocol family 44 [ 0.924976] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.926353] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.928030] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.929695] PCI: CLS 0 bytes, default 64 [ 0.931349] Unpacking initramfs... [ 3.636799] debug: unmapping init [mem 0xffff953ebcc54000-0xffff953ebffbffff] [ 3.660341] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.663974] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.675283] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.107392] hrtimer: interrupt took 8470681 ns [ 6.175610] Initialise system trusted keyrings [ 6.177978] Key type blacklist registered [ 6.184251] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.193600] zbud: loaded [ 6.197568] *** VALIDATE nfs *** [ 6.199116] *** VALIDATE nfs4 *** [ 6.203777] pstore: using deflate compression [ 6.271096] Platform Keyring initialized [ 6.906428] NET: Registered protocol family 38 [ 6.908186] Key type asymmetric registered [ 6.911881] Asymmetric key parser 'x509' registered [ 6.921876] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.924983] io scheduler mq-deadline registered [ 6.930210] io scheduler kyber registered [ 6.944366] io scheduler bfq registered [ 6.957894] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.970781] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.980667] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.987240] ACPI: Power Button [PWRF] [ 7.021309] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.062545] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.101358] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.122822] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.165851] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.230513] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.281754] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.300294] Non-volatile memory driver v1.3 [ 7.301824] Linux agpgart interface v0.103 [ 7.376385] virtio_blk virtio1: [vda] 149992 512-byte logical blocks (76.8 MB/73.2 MiB) [ 7.381415] vda: detected capacity change from 0 to 76795904 [ 7.412307] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.416308] vdb: detected capacity change from 0 to 1073741824 [ 7.436534] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.440333] vdc: detected capacity change from 0 to 2621440000 [ 7.458486] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.462329] vdd: detected capacity change from 0 to 2621440000 [ 7.484826] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.489155] vde: detected capacity change from 0 to 4294967296 [ 7.542071] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.544469] vdf: detected capacity change from 0 to 4294967296 [ 7.552629] libphy: Fixed MDIO Bus: probed [ 7.580939] usbcore: registered new interface driver usbserial_generic [ 7.592955] usbserial: USB Serial support registered for generic [ 7.597110] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.611132] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.616377] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.626531] mousedev: PS/2 mouse device common for all mice [ 7.643213] rtc_cmos 00:05: RTC can wake from S4 [ 7.656470] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.657978] rtc_cmos 00:05: registered as rtc0 [ 7.684667] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.697673] intel_pstate: CPU model not supported [ 7.700693] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.715246] hid: raw HID events driver (C) Jiri Kosina [ 7.731167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.732992] usbcore: registered new interface driver usbhid [ 7.751629] usbhid: USB HID core driver [ 7.754051] drop_monitor: Initializing network drop monitor service [ 7.759334] Initializing XFRM netlink socket [ 7.765547] NET: Registered protocol family 10 [ 7.771552] Segment Routing with IPv6 [ 7.773044] NET: Registered protocol family 17 [ 7.779228] mpls_gso: MPLS GSO support [ 7.807776] RAS: Correctable Errors collector initialized. [ 7.818829] AVX version of gcm_enc/dec engaged. [ 7.828046] AES CTR mode by8 optimization enabled [ 8.065507] sched_clock: Marking stable (8065482410, 0)->(8997480257, -931997847) [ 8.070947] registered taskstats version 1 [ 8.074829] Loading compiled-in X.509 certificates [ 8.078160] zswap: loaded using pool lzo/zbud [ 8.120228] Key type big_key registered [ 8.162943] Key type encrypted registered [ 8.166851] ima: No TPM chip found, activating TPM-bypass! [ 8.175918] ima: Allocated hash algorithm: sha1 [ 8.185949] ima: No architecture policies found [ 8.194052] evm: Initialising EVM extended attributes: [ 8.200749] evm: security.selinux [ 8.205231] evm: security.ima [ 8.208641] evm: security.capability [ 8.210223] evm: HMAC attrs: 0x1 [ 8.212813] rtc_cmos 00:05: setting system clock to 2026-09-26 03:06:27 UTC (1790391987) [ 8.238172] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 8.252614] debug: unmapping init [mem 0xffffffffb6582000-0xffffffffb6858fff] [ 8.265079] Write protecting the kernel read-only data: 28672k [ 8.279208] debug: unmapping init [mem 0xffffffffb4c03000-0xffffffffb4dfffff] [ 8.289851] debug: unmapping init [mem 0xffffffffb5514000-0xffffffffb55fffff] [ 8.445835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.465467] systemd[1]: Detected virtualization kvm. [ 8.483798] systemd[1]: Detected architecture x86-64. [ 8.499273] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.595699] systemd[1]: No hostname configured. [ 8.601351] systemd[1]: Set hostname to . [ 8.606219] random: systemd: uninitialized urandom read (16 bytes read) [ 8.622286] systemd[1]: Initializing machine ID from random generator. [ 8.932378] random: ln: uninitialized urandom read (6 bytes read) [ 9.304627] random: systemd: uninitialized urandom read (16 bytes read) [ 9.316585] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 9.353822] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.366334] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.525572] device-mapper: uevent: version 1.0.3 [ 11.530396] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 14.387934] virtio_net virtio0 ens2: renamed from eth0 [ 14.431764] random: fast init done [ 14.746247] scsi host0: ata_piix [ 14.826117] scsi host1: ata_piix [ 14.832260] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.840739] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.756555] random: crng init done [ 19.770974] random: 7 urandom warning(s) missed due to ratelimiting [ 23.274276] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 25.305956] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.448586] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.109577] SELinux: Disabled at runtime. [ 29.237601] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.258283] systemd[1]: Detected virtualization kvm. [ 29.264721] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 31.442311] systemd[1]: initrd-switch-root.service: Succeeded. [ 31.459593] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 31.505674] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 31.523212] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 31.542682] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 31.585688] systemd[1]: Starting Journal Service... Starting Journal Service... [ 31.636221] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ 32.456422] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 33.856416] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 35.082456] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.191320] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.508145] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.571574] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 40.927917] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 41.837268] NFS: Registering the id_resolver key type [ 41.839810] Key type id_resolver registered [ 41.841619] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg212-server login: [ 101.807930] libcfs: loading out-of-tree module taints kernel. [ 101.896347] Key type ._llcrypt registered [ 101.899880] Key type .llcrypt registered [ 102.037272] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 121.844158] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 123.767679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.792689] alg: No test for adler32 (adler32-zlib) [ 125.287296] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 126.458683] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 128.352188] Key type lgssc registered [ 130.059943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.192867] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 185.838947] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 199.068668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.102797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.393596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.463389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.580392] Lustre: lustre-MDT0000: new disk, initializing [ 200.729300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.754461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.797768] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.592233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.727682] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.772460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.775213] Lustre: Skipped 1 previous similar message [ 219.872233] Lustre: lustre-MDT0001: new disk, initializing [ 219.958125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.004582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.011440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 224.599919] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.291416] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.728428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.972256] Lustre: lustre-OST0000: new disk, initializing [ 238.980186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.993248] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.052444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.817085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.833381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.888701] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.930995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.733801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.897289] Lustre: lustre-OST0001: new disk, initializing [ 258.906946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 258.916056] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 258.989490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.691257] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.727711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 265.738707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 265.784840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 275.378190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.675286] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.167869] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.495335] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing yml_node [ 294.481162] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 296.598461] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 298.842354] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 300.286612] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 25 23:11:18 EDT 2026 [ 313.954520] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 315.202492] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.256114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.258205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.270919] Lustre: Skipped 1 previous similar message [ 332.278599] Lustre: Skipped 3 previous similar messages [ 335.327555] Lustre: server umount lustre-MDT0000 complete [ 338.580635] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392317 with bad export cookie 15374120104658153785 [ 338.588781] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.593937] LustreError: 6507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 342.500657] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.500991] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.503731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.510192] Lustre: Skipped 3 previous similar messages [ 342.539955] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 343.525377] LustreError: 8751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.525621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.557911] Lustre: Skipped 2 previous similar messages [ 344.919344] Lustre: server umount lustre-MDT0001 complete [ 347.780632] Lustre: server umount lustre-OST0000 complete [ 350.913994] Lustre: server umount lustre-OST0001 complete [ 356.266509] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 363.707211] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 394.655604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.814411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.838258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.902417] Lustre: lustre-MDT0000: new disk, initializing [ 394.970124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.005558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 398.463900] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 406.879843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.972177] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.023900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.028958] Lustre: Skipped 1 previous similar message [ 407.117501] Lustre: lustre-MDT0001: new disk, initializing [ 407.218542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.252442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.265046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 410.706258] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 419.157435] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 424.089952] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.672207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.896901] Lustre: lustre-OST0000: new disk, initializing [ 429.900523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.906247] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.981080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.711379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.717733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.855719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.396181] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 443.379661] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.560690] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.725420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 451.664320] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 451.820372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.699970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.713216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.714278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.716643] Lustre: Skipped 1 previous similar message [ 457.733099] Lustre: Skipped 1 previous similar message [ 459.756921] Lustre: server umount lustre-OST0000 complete [ 463.841680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 463.850667] LustreError: Skipped 1 previous similar message [ 463.854917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.871892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.946682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.665445] Lustre: server umount lustre-MDT0000 complete [ 472.512780] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392451 with bad export cookie 15374120104658155318 [ 472.517705] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.525520] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 472.848763] Lustre: server umount lustre-MDT0001 complete [ 475.713288] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:14:14 (1790392454) [ 477.749300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 479.382495] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:14:17 (1790392457) [ 484.837568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.174320] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.193145] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 485.262462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.480750] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 490.469501] LustreError: 19454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.027264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.020793] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 505.427654] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 509.589075] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.346267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.662746] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 528.504629] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 540.130487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.136968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.159429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 540.163442] Lustre: Skipped 2 previous similar messages [ 545.922443] Lustre: server umount lustre-OST0000 complete [ 554.979156] Lustre: server umount lustre-MDT0000 complete [ 556.513073] LustreError: 19449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.531123] LustreError: 19449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 557.909750] LustreError: 19435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392537 with bad export cookie 15374120104658156200 [ 557.918665] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.240266] Lustre: server umount lustre-MDT0001 complete [ 566.303141] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 568.363447] Key type lgssc unregistered [ 568.596118] LNet: 22982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.599490] LNetError: 22982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 568.612604] LNet: Removed LNI 192.168.202.112@tcp [ 569.331454] Key type .llcrypt unregistered [ 569.334152] Key type ._llcrypt unregistered [ 585.032836] Key type ._llcrypt registered [ 585.035975] Key type .llcrypt registered [ 585.114977] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 585.884849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 585.932510] alg: No test for adler32 (adler32-zlib) [ 586.892203] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 587.090274] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 588.759473] Key type lgssc registered [ 589.594126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 595.471705] loop: module loaded [ 595.535341] loop0: detected capacity change from 0 to 204800000 [ 595.725563] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 595.732582] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 595.905256] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.368972] loop0: detected capacity change from 0 to 204800000 [ 600.528981] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.539589] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 600.545733] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 603.769786] loop0: detected capacity change from 0 to 204800000 [ 607.187080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.200781] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.344195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 608.388654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 608.461374] Lustre: lustre-MDT0000: new disk, initializing [ 608.524859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.551987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 612.110335] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 619.270278] loop1: detected capacity change from 0 to 204800000 [ 623.793470] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 624.045550] Lustre: lustre-OST07c6: new disk, initializing [ 624.048834] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 624.051797] Lustre: Skipped 1 previous similar message [ 624.056167] Lustre: 26140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 624.119577] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 628.550270] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 634.457408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 634.462933] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 634.489698] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 637.437450] Lustre: 26552:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 637.448345] Lustre: 26552:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 637.459726] Lustre: 26552:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 637.464184] Lustre: 26552:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 637.467642] Lustre: 26552:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 637.472404] Lustre: 26552:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 663.956192] Lustre: 25130:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 663.961669] Lustre: 25130:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 663.965786] Lustre: 25130:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 663.974252] Lustre: 25130:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 663.981645] Lustre: 25130:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 663.985671] Lustre: 25130:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.336978] Lustre: 26552:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 668.347196] Lustre: 26552:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 668.350979] Lustre: 26552:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 668.355107] Lustre: 26552:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.359902] Lustre: 26552:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 668.364193] Lustre: 26552:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.368222] Lustre: 26552:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 668.372580] Lustre: 26552:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.376907] Lustre: 26552:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 668.381927] Lustre: 26552:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.385111] Lustre: 26552:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 668.389045] Lustre: 26552:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.559133] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 671.573378] Lustre: 25129:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 671.583157] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 671.594947] Lustre: 25129:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.603597] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 671.611789] Lustre: 25129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.618465] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 671.624784] Lustre: 25129:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.628760] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 671.633367] Lustre: 25129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 671.637560] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 671.644424] Lustre: 25129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.781466] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 675.793077] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 675.800703] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 675.806178] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.811563] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 675.816829] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.821168] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 675.825866] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.831298] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 675.837186] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.842716] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 675.848570] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.412501] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 686.421245] Lustre: 25128:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 686.427615] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 686.434746] Lustre: 25128:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.441197] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.449612] Lustre: 25128:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.455553] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 686.465522] Lustre: 25128:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.470413] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.474722] Lustre: 25128:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 686.481058] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.486112] Lustre: 25128:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 736.737193] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 736.742738] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.770748] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 738.963251] Lustre: server umount lustre-OST07c6 complete [ 746.228920] Lustre: server umount lustre-MDT0000 complete [ 758.665464] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 760.816914] Key type lgssc unregistered [ 761.052683] LNet: 28722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.058828] LNetError: 28722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.073683] LNet: Removed LNI 192.168.202.112@tcp [ 761.670725] Key type .llcrypt unregistered [ 761.672150] Key type ._llcrypt unregistered [ 775.214801] Key type ._llcrypt registered [ 775.216683] Key type .llcrypt registered [ 775.291686] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 776.284605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.313056] alg: No test for adler32 (adler32-zlib) [ 777.303966] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 777.502498] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 779.159205] Key type lgssc registered [ 780.015671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 785.584085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 785.600229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.968892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.700519] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 797.053735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.153963] LustreError: 29966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.178191] LustreError: 29966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 797.351666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 800.478341] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 806.217906] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 810.553467] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 815.258078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 815.618975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.658855] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 824.818555] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 831.983947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 847.335054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.338110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 847.338122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 847.344196] Lustre: Skipped 1 previous similar message [ 847.360849] LustreError: Skipped 1 previous similar message [ 852.448570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.456595] Lustre: Skipped 2 previous similar messages [ 852.829783] Lustre: server umount lustre-OST0000 complete [ 857.572877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.576278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 857.594600] Lustre: Skipped 1 previous similar message [ 857.604288] Lustre: Skipped 1 previous similar message [ 862.689284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.703934] Lustre: Skipped 1 previous similar message [ 867.808604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.814727] Lustre: Skipped 1 previous similar message [ 870.371225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 870.538845] Lustre: server umount lustre-MDT0000 complete [ 872.930214] LustreError: 31074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.939561] LustreError: 31074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 873.325401] LustreError: 29948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392852 with bad export cookie 16549818468543112386 [ 873.333276] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.339256] LustreError: 29948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 873.712740] Lustre: server umount lustre-MDT0001 complete [ 880.649952] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 882.356326] Key type lgssc unregistered [ 882.535221] LNet: 33720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.538866] LNetError: 33720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.547201] LNet: Removed LNI 192.168.202.112@tcp [ 883.022804] Key type .llcrypt unregistered [ 883.024961] Key type ._llcrypt unregistered [ 893.530522] Key type ._llcrypt registered [ 893.532208] Key type .llcrypt registered [ 893.587577] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 894.247626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 894.260540] alg: No test for adler32 (adler32-zlib) [ 895.145936] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 895.309466] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 896.936146] Key type lgssc registered [ 897.642309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 902.390697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 902.406853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.759141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.509308] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 911.743136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.996720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 914.417794] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 919.116730] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 922.874587] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 927.285435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 927.584597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.648375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 931.838095] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 937.021572] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 945.123674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 945.130509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.144460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.168196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.172358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.172796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.866807] Lustre: server umount lustre-OST0000 complete [ 957.408651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.412638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 957.416300] Lustre: Skipped 1 previous similar message [ 957.422146] Lustre: Skipped 2 previous similar messages [ 959.325968] Lustre: server umount lustre-MDT0000 complete [ 961.804482] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790392941 with bad export cookie 12773465100764676719 [ 961.808181] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.810770] LustreError: 34947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 961.975809] Lustre: server umount lustre-MDT0001 complete [ 967.809176] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 969.344314] Key type lgssc unregistered [ 969.543228] LNet: 38501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.550294] LNetError: 38501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.560229] LNet: Removed LNI 192.168.202.112@tcp [ 970.084459] Key type .llcrypt unregistered [ 970.086305] Key type ._llcrypt unregistered [ 982.645162] Key type ._llcrypt registered [ 982.646925] Key type .llcrypt registered [ 982.724175] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 983.336334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.358931] alg: No test for adler32 (adler32-zlib) [ 984.298798] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 984.402220] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 985.999190] Key type lgssc registered [ 986.630596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 991.105851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 991.123261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.368445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 994.365770] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 998.902322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.119259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1001.453378] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1005.544293] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1008.538246] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.071323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.263775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.654712] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1017.391658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1019.668898] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.600904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.604352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.605734] Lustre: Skipped 2 previous similar messages [ 1029.610681] Lustre: Skipped 2 previous similar messages [ 1032.544549] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) [ 1034.722534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.726498] Lustre: Skipped 2 previous similar messages [ 1034.942098] Lustre: server umount lustre-MDT0000 complete [ 1037.156613] LustreError: 39874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393016 with bad export cookie 9307461822673857113 [ 1037.161113] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.166309] LustreError: 39874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1037.645441] LustreError: 39888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.661380] LustreError: 39888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1037.666404] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [ 1039.840688] LustreError: 39887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.847740] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.766369] LustreError: 39887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.767968] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [ 1042.783638] Lustre: Skipped 1 previous similar message [ 1043.420424] Lustre: server umount lustre-MDT0001 complete [ 1062.277982] Lustre: server umount lustre-OST0000 complete [ 1071.967574] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1073.475528] Key type lgssc unregistered [ 1073.618594] LNet: 43748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.622260] LNetError: 43748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.632743] LNet: Removed LNI 192.168.202.112@tcp [ 1074.102332] Key type .llcrypt unregistered [ 1074.107565] Key type ._llcrypt unregistered [ 1087.266769] Key type ._llcrypt registered [ 1087.268357] Key type .llcrypt registered [ 1087.347118] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1087.776544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.860029] alg: No test for adler32 (adler32-zlib) [ 1088.775803] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1088.915894] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1090.543409] Key type lgssc registered [ 1091.282143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.864746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.880232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.106473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.556469] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1104.287165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1104.510510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1106.587198] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1110.605521] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1113.379461] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1117.102924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1117.354137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.721976] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1122.475488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1124.692986] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1136.099968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.102690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1136.103344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1136.110540] Lustre: Skipped 1 previous similar message [ 1137.511543] Lustre: lustre-OST0000: Not available for connect from 192.168.202.12@tcp (stopping) [ 1137.515528] Lustre: Skipped 1 previous similar message [ 1139.632535] Lustre: server umount lustre-OST0000 complete [ 1155.553464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1155.559131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.569326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.578911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.605534] Lustre: server umount lustre-MDT0000 complete [ 1161.699827] LustreError: 45134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.713280] LustreError: 45134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1162.744650] LustreError: 45119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393142 with bad export cookie 9824308372509166053 [ 1162.747715] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.751651] LustreError: 45119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1177.055146] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1177.213570] Lustre: server umount lustre-MDT0001 complete [ 1182.831910] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1184.458885] Key type lgssc unregistered [ 1184.633585] LNet: 48845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.640099] LNetError: 48845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.650872] LNet: Removed LNI 192.168.202.112@tcp [ 1185.076817] Key type .llcrypt unregistered [ 1185.079222] Key type ._llcrypt unregistered [ 1197.271882] Key type ._llcrypt registered [ 1197.273324] Key type .llcrypt registered [ 1197.335484] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1197.933623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.948996] alg: No test for adler32 (adler32-zlib) [ 1198.833556] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1198.962470] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1200.551160] Key type lgssc registered [ 1201.311191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1205.441210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1205.448670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.726030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.948306] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1213.912924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.205563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1216.740508] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1221.190739] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1224.689215] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1228.955190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1229.252539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.337546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1233.042577] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1237.046898] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.354720] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1247.241166] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1258.294621] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-2523h27m16s [ 1259.065128] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1260.512265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1260.516425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.523774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.538054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.538460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.542741] Lustre: Skipped 1 previous similar message [ 1261.547771] Lustre: Skipped 2 previous similar messages [ 1262.564858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.567643] Lustre: Skipped 1 previous similar message [ 1264.609575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.614192] Lustre: Skipped 2 previous similar messages [ 1266.363714] Lustre: server umount lustre-MDT0000 complete [ 1266.661614] LustreError: 50232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.667486] LustreError: 50232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1267.681189] LustreError: 51523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.892364] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1268.709984] LustreError: 52725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.720015] LustreError: 52725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1270.754575] LustreError: 51524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.762135] LustreError: 51524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1271.045388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.112785] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1271.246130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.279915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1272.296161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1273.381634] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1275.275728] Lustre: DEBUG MARKER: Wait for df (36676) ... [ 1275.951355] Lustre: DEBUG MARKER: done [ 1277.410994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.414617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1277.417196] Lustre: Skipped 1 previous similar message [ 1277.432864] Lustre: Skipped 4 previous similar messages [ 1283.458785] Lustre: server umount lustre-OST0000 complete [ 1287.651130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1287.651442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1287.653454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1287.653460] Lustre: Skipped 2 previous similar messages [ 1287.657839] Lustre: Skipped 1 previous similar message [ 1291.711119] Lustre: server umount lustre-MDT0000 complete [ 1293.611491] LustreError: 52864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393272 with bad export cookie 10486787694073604740 [ 1293.616291] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.619300] LustreError: 52864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1293.840365] Lustre: server umount lustre-MDT0001 complete [ 1298.418645] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1299.741758] Key type lgssc unregistered [ 1299.907726] LNet: 55148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.916224] LNetError: 55148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.938364] LNet: Removed LNI 192.168.202.112@tcp [ 1300.376651] Key type .llcrypt unregistered [ 1300.378791] Key type ._llcrypt unregistered [ 1319.176636] Key type ._llcrypt registered [ 1319.178417] Key type .llcrypt registered [ 1319.232582] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1319.675645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1319.705336] alg: No test for adler32 (adler32-zlib) [ 1320.619309] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1320.751033] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1322.343170] Key type lgssc registered [ 1322.838816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.376276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.382953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.469150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1327.477151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1327.539385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.130826] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1333.269919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.330665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1333.346712] Lustre: 57331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1333.493401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.347705] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1338.665527] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1341.092294] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1344.174161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.245240] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1344.343250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.745977] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1347.933595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1349.911385] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1353.806688] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-2523h28m52s [ 1354.486450] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1357.971112] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1359.147815] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1359.171809] LustreError: 57335:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1379.678372] Lustre: lustre-MDT0000: Client a20a5717-6ddf-4d2d-a82c-fcb8f66d6bbe (at 192.168.202.12@tcp) reconnecting [ 1399.223120] LustreError: 57335:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1400.224765] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1401.318560] Lustre: DEBUG MARKER: done [ 1404.384802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.385537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.391825] LustreError: Skipped 1 previous similar message [ 1404.392688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.403789] Lustre: Skipped 1 previous similar message [ 1409.436704] Lustre: server umount lustre-OST0000 complete [ 1414.627339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.628948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.631567] Lustre: Skipped 1 previous similar message [ 1414.634691] Lustre: Skipped 1 previous similar message [ 1415.648660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.652442] Lustre: Skipped 1 previous similar message [ 1421.794518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1421.797837] Lustre: Skipped 1 previous similar message [ 1425.375144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1425.467756] Lustre: server umount lustre-MDT0000 complete [ 1427.453194] LustreError: 57322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393406 with bad export cookie 346548638951778203 [ 1427.454626] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1427.463300] LustreError: 57322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1427.640898] Lustre: server umount lustre-MDT0001 complete [ 1432.739190] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1434.186323] Key type lgssc unregistered [ 1434.343384] LNet: 61654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.346511] LNetError: 61654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.359510] LNet: Removed LNI 192.168.202.112@tcp [ 1434.781106] Key type .llcrypt unregistered [ 1434.783053] Key type ._llcrypt unregistered [ 1456.588744] Key type ._llcrypt registered [ 1456.590268] Key type .llcrypt registered [ 1456.645407] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1457.058299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.067188] alg: No test for adler32 (adler32-zlib) [ 1457.952582] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1458.064169] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1459.687866] Key type lgssc registered [ 1460.317639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1461.910128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1461.920393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.072100] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1463.082539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1463.178480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.737217] Lustre: Failing over lustre-MDT0000 [ 1463.897835] Lustre: server umount lustre-MDT0000 complete [ 1467.419907] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:30:46 (1790393446) [ 1470.455294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.635738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.645259] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1477.123843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.182548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1477.195221] Lustre: 64422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1477.326113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1479.423721] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1483.095863] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1485.767442] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1489.036104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.153870] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1489.259565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.252457] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1495.729714] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1497.135722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1504.186946] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1507.297847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.298748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.302346] Lustre: Skipped 1 previous similar message [ 1507.305683] Lustre: Skipped 2 previous similar messages [ 1512.418050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1512.421294] Lustre: Skipped 2 previous similar messages [ 1512.817879] Lustre: server umount lustre-MDT0000 complete [ 1514.473359] LustreError: 64415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393493 with bad export cookie 16387962961694456082 [ 1514.475561] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1514.478276] LustreError: 64415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1514.611679] Lustre: server umount lustre-MDT0001 complete [ 1516.088956] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1518.000350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1521.881273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.999113] LustreError: 68177:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1524.007808] LustreError: 68177:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000e5cb7d68 x1877363588629632/t0(0) o250->MGC192.168.202.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1790393503 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1524.023253] LustreError: 68177:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1524.706362] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe36dbd7c8c0e81ee [ 1524.711448] Lustre: MGC192.168.202.112@tcp: Connection restored to 0@lo (at 0@lo) [ 1524.897400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.943710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1526.791796] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1530.341502] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.344685] LustreError: 66272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1530.347644] Lustre: Skipped 1 previous similar message [ 1530.362548] LustreError: 66272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.728471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.737563] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1532.959124] Lustre: 63348:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790393496/real 1790393496] req@0000000091272ff7 x1877363588629888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790393512 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1535.007192] Lustre: 63346:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790393498/real 1790393498] req@00000000e93aea19 x1877363588630016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790393514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1535.840630] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1535.973461] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1537.003296] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1538.015162] Lustre: 63347:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790393501/real 1790393501] req@00000000bf69acf4 x1877363588630144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790393517 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1538.246780] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1540.063455] Lustre: 63348:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790393503/real 1790393503] req@0000000016b2f320 x1877363588630400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790393519 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1541.088597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1541.092269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.098373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.645185] Lustre: server umount lustre-MDT0000 complete [ 1547.232771] LustreError: 68192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.238748] LustreError: 68192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1548.241599] LustreError: 64455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393527 with bad export cookie 16387962961694458350 [ 1548.244700] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1548.254169] LustreError: 64455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1548.373474] Lustre: server umount lustre-MDT0001 complete [ 1551.100078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1555.100847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.495869] LustreError: 63345:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001aa528f0 x1877363588650880/t0(0) o250->MGC192.168.202.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1558.632489] LustreError: 66271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.672560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.675768] Lustre: Skipped 1 previous similar message [ 1558.718532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1560.516513] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1564.128555] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.129316] LustreError: 68213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.134676] Lustre: Skipped 2 previous similar messages [ 1564.459258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.396366] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1567.781892] Lustre: 63347:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790393531/real 1790393531] req@00000000c0d6417b x1877363588650496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790393547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1569.405621] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1569.768579] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1571.898397] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1574.881027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.885046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.894057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.900618] Lustre: Skipped 3 previous similar messages [ 1580.730250] Lustre: server umount lustre-MDT0000 complete [ 1582.055560] LustreError: 66271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1582.061735] LustreError: 66271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1582.383465] LustreError: 64455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393561 with bad export cookie 16387962961694459729 [ 1582.386455] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.389024] LustreError: 64455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1582.524120] Lustre: server umount lustre-MDT0001 complete [ 1585.582369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1586.918327] Lustre: DEBUG MARKER: files compared the same [ 1596.979193] Lustre: server umount lustre-OST0000 complete [ 1604.546468] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1606.066400] Key type lgssc unregistered [ 1606.253576] LNet: 74140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.256397] LNetError: 74140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.271071] LNet: Removed LNI 192.168.202.112@tcp [ 1607.775133] Key type .llcrypt unregistered [ 1607.777581] Key type ._llcrypt unregistered [ 1619.505735] Key type ._llcrypt registered [ 1619.507409] Key type .llcrypt registered [ 1619.575589] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1620.089512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.108146] alg: No test for adler32 (adler32-zlib) [ 1621.050619] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1621.183653] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1622.783511] Key type lgssc registered [ 1623.418631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1627.428956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1627.441946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.669425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.829628] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug +malloc all [ 1635.169557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.327462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.214385] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug +malloc all [ 1640.475567] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1643.187644] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1646.637177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1646.802207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.710795] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug +malloc all [ 1652.972468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1653.415841] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1668.575465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1668.578896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.583266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.623842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1670.623925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1670.629167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.465674] Lustre: server umount lustre-OST0000 complete [ 1675.744207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.753706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.757538] Lustre: Skipped 2 previous similar messages [ 1680.149686] Lustre: server umount lustre-MDT0000 complete [ 1680.863600] LustreError: 75530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.869757] LustreError: 75530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1681.544723] LustreError: 76265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393660 with bad export cookie 7515649242864834740 [ 1681.546162] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1681.550735] LustreError: 76265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1681.677207] Lustre: server umount lustre-MDT0001 complete [ 1685.675768] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1686.953712] Key type lgssc unregistered [ 1687.124495] LNet: 79080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.127816] LNetError: 79080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.138227] LNet: Removed LNI 192.168.202.112@tcp [ 1687.539899] Key type .llcrypt unregistered [ 1687.541314] Key type ._llcrypt unregistered [ 1697.744203] Key type ._llcrypt registered [ 1697.745688] Key type .llcrypt registered [ 1697.800437] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1698.145313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.188828] alg: No test for adler32 (adler32-zlib) [ 1699.066456] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1699.175580] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1700.767206] Key type lgssc registered [ 1701.223440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.255181] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.263988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.711474] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1724.831383] LustreError: 80443:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.112@tcp: failed processing log, type 4: rc = -110 [ 1751.455195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.864282] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1756.661975] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1759.435762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.482138] Lustre: *** cfs_fail_loc=706, val=0*** [ 1759.613714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.642140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1761.068817] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1764.222623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.367399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1765.988728] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1768.553588] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1770.814434] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1774.561482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1774.564500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.569425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1775.077670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1775.081106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.088157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.609422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1776.612384] Lustre: Skipped 1 previous similar message [ 1781.730420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.732583] Lustre: Skipped 1 previous similar message [ 1786.335104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1786.419331] Lustre: server umount lustre-OST0000 complete [ 1789.919865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1789.922626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.926887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.968902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.207117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1802.208718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.213177] Lustre: Skipped 5 previous similar messages [ 1802.270863] Lustre: server umount lustre-MDT0000 complete [ 1803.648341] LustreError: 80451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790393782 with bad export cookie 17152986856351721377 [ 1803.651474] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.654014] LustreError: 80451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1803.782114] Lustre: server umount lustre-MDT0001 complete [ 1807.810149] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1808.970150] Key type lgssc unregistered [ 1809.112605] LNet: 84056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.115303] LNetError: 84056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.122567] LNet: Removed LNI 192.168.202.112@tcp [ 1809.478367] Key type .llcrypt unregistered [ 1809.480155] Key type ._llcrypt unregistered [ 1820.387971] Key type ._llcrypt registered [ 1820.389272] Key type .llcrypt registered [ 1820.429636] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1820.771715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1820.825986] alg: No test for adler32 (adler32-zlib) [ 1821.700534] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 1821.803680] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 1823.399142] Key type lgssc registered [ 1823.827848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.232246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.242391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.808851] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1833.166111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.388075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.045800] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1838.559680] LustreError: 86133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1838.566169] LustreError: 86133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1838.753941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.913686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.430416] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1843.039969] LustreError: 86137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.047685] LustreError: 86137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1845.056984] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.062044] Lustre: MGS: Client 8407a631-d57a-45ba-88f7-430057aaa95e (at 0@lo) reconnecting [ 1845.161995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.190767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1846.751611] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1855.458487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.463785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.464149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.469605] Lustre: Skipped 1 previous similar message [ 1860.577302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1860.579557] Lustre: Skipped 1 previous similar message [ 1861.012540] Lustre: server umount lustre-OST0000 complete [ 1865.696880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.697147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.704814] Lustre: Skipped 1 previous similar message [ 1865.706813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.709046] Lustre: Skipped 1 previous similar message [ 1868.483708] Lustre: server umount lustre-MDT0000 complete [ 1870.347983] Lustre: server umount lustre-MDT0001 complete [ 1871.732690] Lustre: server umount MGS complete [ 1875.943398] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:37:35 (1790393855) [ 1887.123671] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 1890.711105] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 1903.960416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.442145] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1909.755801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.840839] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1909.859488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1909.895946] Lustre: lustre-MDT0001: new disk, initializing [ 1909.923324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.932368] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1909.935459] Lustre: 93290:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1909.938791] LustreError: 93290:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1911.393986] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1915.825591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.940536] Lustre: lustre-OST0000: new disk, initializing [ 1915.942308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1915.946512] Lustre: 94055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1915.979389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.029307] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1921.968795] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1923.615831] Lustre: 94814:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000770c11f5 x1877363970311552/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790393918 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1924.591000] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1924.595525] Lustre: MGS: Client c221e21f-1307-4597-a88c-35f11d129749 (at 0@lo) reconnecting [ 1924.612872] Lustre: 95100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1924.625710] LustreError: 93275:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1924.629247] LustreError: 93275:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1924.635230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1924.678314] Lustre: lustre-MDT0000: new disk, initializing [ 1924.730191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.741244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1925.792174] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1925.792192] Lustre: 95116:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000bbf84287 x1877363970317056/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790393921 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1925.801869] Lustre: 95116:0:(client.c:1652:after_reply()) Skipped 1 previous similar message [ 1926.267207] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 1930.850891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1930.854350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1930.857446] Lustre: Skipped 1 previous similar message [ 1930.863436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1951.202485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.205898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.206560] Lustre: Skipped 2 previous similar messages [ 1956.320775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1956.323123] Lustre: Skipped 2 previous similar messages [ 1956.501973] Lustre: server umount lustre-OST0000 complete [ 1961.440679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1961.441175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.446467] Lustre: Skipped 1 previous similar message [ 1966.560037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.562646] Lustre: Skipped 3 previous similar messages [ 1972.191126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1972.240763] Lustre: server umount lustre-MDT0000 complete [ 1979.767338] Lustre: server umount lustre-MDT0001 complete [ 1981.284525] Lustre: server umount MGS complete [ 1985.684278] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:39:24 (1790393964) [ 1992.421234] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 1993.554543] Key type lgssc unregistered [ 1993.691524] LNet: 97621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.694307] LNetError: 97621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.701318] LNet: Removed LNI 192.168.202.112@tcp [ 1994.038469] Key type .llcrypt unregistered [ 1994.039927] Key type ._llcrypt unregistered [ 2001.257136] Key type ._llcrypt registered [ 2001.258117] Key type .llcrypt registered [ 2001.296983] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2001.629739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.656068] alg: No test for adler32 (adler32-zlib) [ 2002.508539] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2002.594100] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2004.175172] Key type lgssc registered [ 2004.556897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.338303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2007.345564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.484694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.888741] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2012.807906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.919439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.236154] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2016.820893] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2018.755294] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2021.078117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.197749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.145811] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2025.679701] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2033.510892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2038.754427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.754889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2038.756222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2038.758892] Lustre: Skipped 1 previous similar message [ 2038.761296] LustreError: Skipped 1 previous similar message [ 2043.875328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2043.879387] Lustre: Skipped 2 previous similar messages [ 2048.997673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.000303] Lustre: Skipped 1 previous similar message [ 2053.087145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2053.142291] Lustre: server umount lustre-OST0000 complete [ 2059.232704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2059.233371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.234186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.241986] Lustre: Skipped 1 previous similar message [ 2060.573735] Lustre: server umount lustre-MDT0000 complete [ 2061.955221] LustreError: 98795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394041 with bad export cookie 3457028642764330097 [ 2061.957742] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.959400] LustreError: 98795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2062.068852] Lustre: server umount lustre-MDT0001 complete [ 2065.673592] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2066.800032] Key type lgssc unregistered [ 2066.938459] LNet: 102366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.941274] LNetError: 102366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.954366] LNet: Removed LNI 192.168.202.112@tcp [ 2067.306613] Key type .llcrypt unregistered [ 2067.307956] Key type ._llcrypt unregistered [ 2074.304431] Key type ._llcrypt registered [ 2074.305561] Key type .llcrypt registered [ 2074.351457] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2074.792886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.818744] alg: No test for adler32 (adler32-zlib) [ 2075.666055] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2075.755464] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2077.335183] Key type lgssc registered [ 2077.708740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2079.940330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2079.940433] LustreError: 103425:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2079.943048] Lustre: Skipped 1 previous similar message [ 2079.943058] LustreError: 103424:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2079.949934] LustreError: 103425:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2079.952748] LustreError: 103425:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2079.956196] LustreError: 103425:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2079.956575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.103416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.672219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.763157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.215514] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2086.991888] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing lsmod [ 2087.983100] LustreError: 104629:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 2087.983196] LustreError: 104628:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 2087.988579] LustreError: 104629:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2087.991389] LustreError: 104629:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2087.996172] LustreError: 104629:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2087.997203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.128491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.537053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.538648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.543014] Lustre: Skipped 2 previous similar messages [ 2093.545166] Lustre: Skipped 1 previous similar message [ 2094.562446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.564627] Lustre: Skipped 2 previous similar messages [ 2095.195501] Lustre: server umount lustre-MDT0000 complete [ 2096.591666] LustreError: 103487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394075 with bad export cookie 1001976040248594489 [ 2096.594759] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.596755] LustreError: 103487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.607512] LustreError: 103893:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000363cf570 x1877364236631936/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.706182] Lustre: server umount lustre-MDT0001 complete [ 2107.950571] Lustre: server umount lustre-OST0000 complete [ 2111.208664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2111.363389] LustreError: 105695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.369635] LustreError: 105695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2111.390244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2112.743328] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2115.764839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.896059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2117.317062] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2119.924832] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2121.813241] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.248315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.371777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.333984] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2128.929709] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2136.678218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2142.180853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.181142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.182817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.193237] Lustre: Skipped 1 previous similar message [ 2146.963523] Lustre: server umount lustre-OST0000 complete [ 2151.904247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.907152] LustreError: Skipped 1 previous similar message [ 2151.908619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.912704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.914880] Lustre: Skipped 1 previous similar message [ 2154.330680] Lustre: server umount lustre-MDT0000 complete [ 2155.520238] LustreError: 105677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394134 with bad export cookie 1001976040248595175 [ 2155.523748] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.523941] LustreError: 105677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2155.634073] Lustre: server umount lustre-MDT0001 complete [ 2159.107624] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2160.123879] Key type lgssc unregistered [ 2160.251362] LNet: 109245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.254289] LNetError: 109245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.263319] LNet: Removed LNI 192.168.202.112@tcp [ 2160.544260] Key type .llcrypt unregistered [ 2160.545483] Key type ._llcrypt unregistered [ 2169.495830] Key type ._llcrypt registered [ 2169.496989] Key type .llcrypt registered [ 2169.532804] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2169.851804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.877098] alg: No test for adler32 (adler32-zlib) [ 2170.747260] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2170.842578] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2172.423199] Key type lgssc registered [ 2172.845080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2175.884256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2175.892946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.055224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.656657] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2181.810539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.931982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.411713] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2186.161877] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2188.118208] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2190.699879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2190.853867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.103469] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2195.714942] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2202.147637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2207.997406] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2208.815583] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2212.320659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.321225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.324675] LustreError: Skipped 1 previous similar message [ 2212.329271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2212.332226] Lustre: Skipped 1 previous similar message [ 2215.953749] Lustre: server umount lustre-OST0000 complete [ 2217.440432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.441137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2217.445206] Lustre: Skipped 2 previous similar messages [ 2222.560193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.563366] Lustre: Skipped 2 previous similar messages [ 2223.384213] Lustre: server umount lustre-MDT0000 complete [ 2224.709313] LustreError: 110616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394203 with bad export cookie 9409056644186731163 [ 2224.712390] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.713813] LustreError: 110616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2224.837061] Lustre: server umount lustre-MDT0001 complete [ 2228.649128] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2229.778375] Key type lgssc unregistered [ 2229.914404] LNet: 114286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.917692] LNetError: 114286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.926355] LNet: Removed LNI 192.168.202.112@tcp [ 2230.247628] Key type .llcrypt unregistered [ 2230.249149] Key type ._llcrypt unregistered [ 2237.635469] Key type ._llcrypt registered [ 2237.637104] Key type .llcrypt registered [ 2237.688596] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2238.025927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.033894] alg: No test for adler32 (adler32-zlib) [ 2238.910224] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2239.014597] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2240.599157] Key type lgssc registered [ 2241.002747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2243.796824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.803961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.987803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.488940] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2249.834770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.988509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2251.461859] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2254.048758] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2256.181449] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2258.686039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.820691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.099742] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2263.833747] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2270.120703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2280.419381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2280.419646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.422571] LustreError: Skipped 1 previous similar message [ 2280.429522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2280.433548] Lustre: Skipped 1 previous similar message [ 2283.551966] Lustre: server umount lustre-OST0000 complete [ 2285.536773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.537632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.540949] Lustre: Skipped 2 previous similar messages [ 2290.656940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.659920] Lustre: Skipped 2 previous similar messages [ 2295.776438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.779142] Lustre: Skipped 1 previous similar message [ 2299.359214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2299.429837] Lustre: server umount lustre-MDT0000 complete [ 2299.872591] LustreError: 115480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.878740] LustreError: 115480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2301.039224] LustreError: 115460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394280 with bad export cookie 15661179770318158512 [ 2301.040989] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.044191] LustreError: 115460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2301.181289] Lustre: server umount lustre-MDT0001 complete [ 2305.491408] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2306.658215] Key type lgssc unregistered [ 2306.802425] LNet: 119032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.806346] LNetError: 119032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.816325] LNet: Removed LNI 192.168.202.112@tcp [ 2307.171096] Key type .llcrypt unregistered [ 2307.172365] Key type ._llcrypt unregistered [ 2317.975350] Key type ._llcrypt registered [ 2317.976728] Key type .llcrypt registered [ 2318.032847] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2318.441201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.476591] alg: No test for adler32 (adler32-zlib) [ 2319.369331] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2319.467296] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2321.055203] Key type lgssc registered [ 2321.497338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2324.965916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2324.975597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.186534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.979665] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2331.885996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.080398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.919417] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2336.882966] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2339.087233] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2341.892115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2342.062446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.305816] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2347.083108] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2351.271335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2354.891032] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2356.310373] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2356.312922] Lustre: Skipped 2 previous similar messages [ 2357.905790] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2357.909382] Lustre: Skipped 2 previous similar messages [ 2362.366991] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2362.378537] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2365.603847] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.12@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2365.607242] Lustre: Skipped 2 previous similar messages [ 2367.610953] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.12@tcp 192.168.0.[2,10]@tcp [ 2369.777989] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.12/24@tcp in log lustre-MDT0000 [ 2369.780565] Lustre: Skipped 2 previous similar messages [ 2375.676536] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.12/24@tcp [ 2375.678598] Lustre: Skipped 1 previous similar message [ 2379.231969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2379.234723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.239504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.305994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.306717] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.307984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.216301] Lustre: server umount lustre-OST0000 complete [ 2387.424862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.425536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.429731] Lustre: Skipped 1 previous similar message [ 2387.432047] Lustre: Skipped 1 previous similar message [ 2391.777248] Lustre: server umount lustre-MDT0000 complete [ 2392.544198] LustreError: 120423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.550836] LustreError: 120423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2393.200182] LustreError: 122260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394372 with bad export cookie 14885934483010061442 [ 2393.203429] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.204850] LustreError: 122260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2393.318939] Lustre: server umount lustre-MDT0001 complete [ 2397.281827] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2398.396891] Key type lgssc unregistered [ 2398.533339] LNet: 125495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.536385] LNetError: 125495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.545411] LNet: Removed LNI 192.168.202.112@tcp [ 2398.906434] Key type .llcrypt unregistered [ 2398.908169] Key type ._llcrypt unregistered [ 2410.292437] Key type ._llcrypt registered [ 2410.293769] Key type .llcrypt registered [ 2410.331992] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2410.669666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.702057] alg: No test for adler32 (adler32-zlib) [ 2411.575459] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2411.678494] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2413.263178] Key type lgssc registered [ 2413.714385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2416.685614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2416.692170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.857539] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2417.861148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.866919] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.12/24@tcp [ 2419.449139] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2422.622672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.735043] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2422.744360] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.12/24@tcp [ 2422.760367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2424.413045] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2427.147316] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2429.299491] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2431.807870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2431.944524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.191775] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2436.912422] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2443.242298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2448.359965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.360762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.363072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.363504] LustreError: Skipped 1 previous similar message [ 2448.376131] Lustre: Skipped 1 previous similar message [ 2452.827477] Lustre: server umount lustre-OST0000 complete [ 2458.592916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.593042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2458.593558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.593563] Lustre: Skipped 1 previous similar message [ 2458.597426] Lustre: Skipped 1 previous similar message [ 2460.570197] Lustre: server umount lustre-MDT0000 complete [ 2461.890389] LustreError: 127009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394441 with bad export cookie 2549703103402351895 [ 2461.893380] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2461.897200] LustreError: 127009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2462.018129] Lustre: server umount lustre-MDT0001 complete [ 2465.866992] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2467.066045] Key type lgssc unregistered [ 2467.198450] LNet: 130612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.203996] LNetError: 130612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.213548] LNet: Removed LNI 192.168.202.112@tcp [ 2467.557365] Key type .llcrypt unregistered [ 2467.558639] Key type ._llcrypt unregistered [ 2477.529471] Key type ._llcrypt registered [ 2477.530756] Key type .llcrypt registered [ 2477.571278] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 2484.203958] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2484.639344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2484.647805] alg: No test for adler32 (adler32-zlib) [ 2485.538692] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2485.656046] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2487.263201] Key type lgssc registered [ 2487.706997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.779243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.786779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.888926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2501.904407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2501.943096] Lustre: lustre-MDT0000: new disk, initializing [ 2501.973969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.982176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2503.739917] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2508.898910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.952207] Lustre: 133979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2509.005327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2509.007885] Lustre: Skipped 1 previous similar message [ 2509.058720] Lustre: lustre-MDT0001: new disk, initializing [ 2509.109978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2509.128323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2509.133977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2510.690886] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2514.741169] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2516.731057] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.101548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.203578] Lustre: lustre-OST0000: new disk, initializing [ 2519.205803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2519.209290] Lustre: 135945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2519.242814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.339628] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2525.148712] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2526.633406] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2526.714774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2526.718694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2526.757503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2527.728424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2529.242312] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2529.332020] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2531.809602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2531.810869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.813780] LustreError: Skipped 1 previous similar message [ 2531.819202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.823091] Lustre: Skipped 1 previous similar message [ 2536.212810] Lustre: server umount lustre-OST0000 complete [ 2539.999961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2540.003629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.008859] Lustre: Skipped 1 previous similar message [ 2540.010849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2542.048031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2542.048912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.643508] Lustre: server umount lustre-MDT0000 complete [ 2544.960221] LustreError: 133972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394524 with bad export cookie 15604500886351765579 [ 2544.963247] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2544.964731] LustreError: 133972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2545.081877] Lustre: server umount lustre-MDT0001 complete [ 2548.078936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.234050] LustreError: 138165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.241991] LustreError: 138165:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2548.258069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.592299] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2552.444549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.579707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.963531] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2556.564455] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2558.582720] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2561.102945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2561.229756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2563.239069] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2565.765063] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2567.237975] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2574.532339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2576.196570] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2576.286629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2579.894355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2579.967361] Lustre: lustre-OST0001: new disk, initializing [ 2579.969812] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2579.973063] Lustre: 141585:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2580.016193] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2580.077309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2580.080278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2580.124751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2582.072316] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2585.543158] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2585.622754] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2587.150935] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2587.235994] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2589.229221] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ee35046c800.ost_server_uuid 50 [ 2589.846313] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ee35046c800.ost_server_uuid in FULL state after 0 sec [ 2595.808742] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2595.811402] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.815672] Lustre: Skipped 1 previous similar message [ 2595.816083] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2595.819432] Lustre: Skipped 2 previous similar messages [ 2598.020931] Lustre: server umount lustre-OST0001 complete [ 2600.929318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.930896] LustreError: 139986:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.930921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2600.934412] Lustre: Skipped 1 previous similar message [ 2600.939835] LustreError: 139986:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2605.527758] Lustre: server umount lustre-OST0000 complete [ 2609.119969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2609.123113] LustreError: Skipped 1 previous similar message [ 2609.125128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.131211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.133450] Lustre: Skipped 1 previous similar message [ 2613.024398] Lustre: server umount lustre-MDT0000 complete [ 2614.612386] LustreError: 141583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394593 with bad export cookie 15604500886351766461 [ 2614.616431] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2614.620890] LustreError: 141583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2614.800369] Lustre: server umount lustre-MDT0001 complete [ 2622.448906] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2623.575806] Key type lgssc unregistered [ 2623.712583] LNet: 144580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.715552] LNetError: 144580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.723363] LNet: Removed LNI 192.168.202.112@tcp [ 2624.114835] Key type .llcrypt unregistered [ 2624.116896] Key type ._llcrypt unregistered [ 2637.972936] Key type ._llcrypt registered [ 2637.973990] Key type .llcrypt registered [ 2638.012997] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 2644.114316] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2644.440437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.470029] alg: No test for adler32 (adler32-zlib) [ 2645.333411] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2645.425334] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2647.007155] Key type lgssc registered [ 2647.386727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2658.791840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2658.800267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.889702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2659.901711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2659.940144] Lustre: lustre-MDT0000: new disk, initializing [ 2659.963170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.970336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2661.291140] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2665.388933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.430333] Lustre: 148701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2665.477473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2665.479259] Lustre: Skipped 1 previous similar message [ 2665.515855] Lustre: lustre-MDT0001: new disk, initializing [ 2665.550097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2665.561315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2665.564577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2666.915206] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2670.712996] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2672.630157] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2675.007771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2675.124233] Lustre: lustre-OST0000: new disk, initializing [ 2675.126094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2675.128920] Lustre: 150665:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2675.158188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.185024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2677.188309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2677.212212] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2677.224839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2681.094393] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2683.115367] Lustre: Failing over lustre-OST0000 [ 2683.171742] Lustre: server umount lustre-OST0000 complete [ 2696.083805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.213090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.217487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.225604] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2697.554200] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2698.211539] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2699.513885] Lustre: Failing over lustre-MDT0000 [ 2699.622020] Lustre: server umount lustre-MDT0000 complete [ 2701.281039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2701.281363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.287796] Lustre: Skipped 2 previous similar messages [ 2703.327205] Lustre: 146930:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790394666/real 1790394666] req@00000000711e2dd1 x1877364833282816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1790394682 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2706.400301] LustreError: 148710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2706.405711] LustreError: 148710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2707.786959] LustreError: 148706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.447193] Lustre: 146929:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790394671/real 1790394671] req@0000000060c9a98b x1877364833284352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790394687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2711.520557] LustreError: 149886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.526339] LustreError: 149886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2712.536563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.580126] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2712.585125] Lustre: 148703:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.112@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2712.663418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2712.683306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.707860] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2712.907032] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2714.015432] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2718.186460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2718.194170] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 2718.215631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2723.297557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.299081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2723.301149] Lustre: Skipped 1 previous similar message [ 2723.303411] LustreError: Skipped 1 previous similar message [ 2723.303954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2723.307880] Lustre: Skipped 1 previous similar message [ 2725.265770] Lustre: server umount lustre-OST0000 complete [ 2728.416919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.417083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.417811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.426933] Lustre: Skipped 1 previous similar message [ 2732.640569] Lustre: server umount lustre-MDT0000 complete [ 2733.535980] LustreError: 148711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.541149] LustreError: 148711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2733.912197] LustreError: 153563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394713 with bad export cookie 11951714701654644210 [ 2733.914953] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2733.916289] LustreError: 153563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2734.040423] Lustre: server umount lustre-MDT0001 complete [ 2737.540990] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 2738.692140] Key type lgssc unregistered [ 2738.830511] LNet: 154312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2738.833547] LNetError: 154312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2738.841290] LNet: Removed LNI 192.168.202.112@tcp [ 2739.169962] Key type .llcrypt unregistered [ 2739.171596] Key type ._llcrypt unregistered [ 2759.429230] Key type ._llcrypt registered [ 2759.430220] Key type .llcrypt registered [ 2759.466868] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2759.800029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2759.856168] alg: No test for adler32 (adler32-zlib) [ 2760.713448] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 2760.803249] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 2762.383170] Key type lgssc registered [ 2762.766622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.269610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.275774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.425820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.816802] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2771.411375] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2773.410093] LustreError: 155855:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2773.413623] LustreError: 155855:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2773.416438] LustreError: 155855:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2773.532141] Lustre: server umount lustre-MDT0000 complete [ 2778.475980] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 2781.334289] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2792.695596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.779669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2792.788806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2792.824083] Lustre: lustre-MDT0000: new disk, initializing [ 2792.843615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.849564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2794.176252] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2798.388653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.430466] Lustre: 159151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2798.453853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2798.456431] Lustre: Skipped 1 previous similar message [ 2798.497405] Lustre: lustre-MDT0001: new disk, initializing [ 2798.530837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2798.543658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2798.546886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2799.935224] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2803.695121] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2805.487127] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2807.773125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2807.902132] Lustre: lustre-OST0000: new disk, initializing [ 2807.903561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2807.905868] Lustre: 161120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2807.936657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.868450] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2813.661848] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2814.970066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2814.974194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2815.012967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2820.524544] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2821.039497] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:53:20 (1790394800) [ 2886.623590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.624068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.627552] Lustre: Skipped 2 previous similar messages [ 2886.631033] Lustre: Skipped 1 previous similar message [ 2891.745142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.747990] Lustre: Skipped 2 previous similar messages [ 2892.451114] Lustre: server umount lustre-MDT0000 complete [ 2893.809639] LustreError: 159144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394873 with bad export cookie 10490649275782937602 [ 2893.810727] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2893.814913] LustreError: 159144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2896.863532] LustreError: 159159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2896.863598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.869245] LustreError: 159159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2896.876046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2898.911505] LustreError: 159158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.911579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2904.031603] LustreError: 160771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2904.031683] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2908.127236] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2908.213048] Lustre: server umount lustre-MDT0001 complete [ 2915.662717] Lustre: server umount lustre-OST0000 complete [ 2918.757457] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 2921.677200] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 2932.956511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.049050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2933.059808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2933.099257] Lustre: lustre-MDT0000: new disk, initializing [ 2933.121529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.128259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2934.534040] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2938.795770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.840620] Lustre: 165748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2938.864119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2938.866096] Lustre: Skipped 1 previous similar message [ 2938.904809] Lustre: lustre-MDT0001: new disk, initializing [ 2938.947598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2938.951118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2940.295452] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2944.103285] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2945.860073] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2948.021331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2948.124643] Lustre: lustre-OST0000: new disk, initializing [ 2948.126304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2948.129238] Lustre: 167716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2948.153508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2948.155263] Lustre: Skipped 1 previous similar message [ 2949.970723] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 2950.201866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2950.205554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2950.247535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2953.900385] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2955.333727] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2955.403182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2956.804636] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2956.884183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2960.354104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2960.355137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.357517] LustreError: Skipped 1 previous similar message [ 2960.363759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2960.366124] Lustre: Skipped 1 previous similar message [ 2963.664833] Lustre: server umount lustre-OST0000 complete [ 2965.473752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.477449] Lustre: Skipped 2 previous similar messages [ 2979.295160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2979.355934] Lustre: server umount lustre-MDT0000 complete [ 2979.808401] LustreError: 165755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.813535] LustreError: 165755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2980.576531] LustreError: 165741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790394959 with bad export cookie 10490649275783096418 [ 2980.581602] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.697193] Lustre: server umount lustre-MDT0001 complete [ 2983.547078] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:56:02 (1790394962) [ 2988.798596] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 2991.862319] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3003.588072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3003.685551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3003.697992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3003.754695] Lustre: lustre-MDT0000: new disk, initializing [ 3003.778669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.785359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3005.224465] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3009.325261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3009.371574] Lustre: 172465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3009.468791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3010.801765] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3014.651228] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3016.554367] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3018.882267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.011807] Lustre: 174441:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3020.348034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3020.395560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3021.129689] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3024.914188] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3030.497853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.498207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3030.501811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3030.506184] Lustre: Skipped 7 previous similar messages [ 3032.916373] Lustre: server umount lustre-OST0000 complete [ 3035.103780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3040.345673] Lustre: server umount lustre-MDT0000 complete [ 3040.735924] LustreError: 173283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.741552] LustreError: 173283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3041.551718] LustreError: 172460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395020 with bad export cookie 10490649275783097286 [ 3041.554719] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.556085] LustreError: 172460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3041.674811] Lustre: server umount lustre-MDT0001 complete [ 3044.275773] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:57:03 (1790395023) [ 3049.239825] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 3052.139693] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3063.388908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.489746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3063.500121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3063.501869] Lustre: Skipped 3 previous similar messages [ 3063.543638] Lustre: lustre-MDT0000: new disk, initializing [ 3063.545367] Lustre: Skipped 2 previous similar messages [ 3063.566929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3063.569181] Lustre: Skipped 2 previous similar messages [ 3063.574371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3063.578493] Lustre: Skipped 2 previous similar messages [ 3064.913706] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3068.989687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.033673] Lustre: 178751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3069.131892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3070.524289] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3074.279580] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3076.118698] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3078.344496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3078.459337] Lustre: 180727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3079.783163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3080.424497] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3084.011122] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3089.892061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.895025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.895369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.895377] Lustre: Skipped 2 previous similar messages [ 3089.895701] Lustre: Skipped 4 previous similar messages [ 3091.980520] Lustre: server umount lustre-OST0000 complete [ 3095.008832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3099.285652] Lustre: server umount lustre-MDT0000 complete [ 3100.128158] LustreError: 179559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.133415] LustreError: 179559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3100.393232] LustreError: 179553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395079 with bad export cookie 10490649275783098357 [ 3100.396582] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3100.396652] LustreError: 179553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3100.511067] Lustre: server umount lustre-MDT0001 complete [ 3104.023406] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 3105.057844] Key type lgssc unregistered [ 3105.189490] LNet: 182584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3105.191822] LNetError: 182584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3105.200375] LNet: Removed LNI 192.168.202.112@tcp [ 3105.497873] Key type .llcrypt unregistered [ 3105.498842] Key type ._llcrypt unregistered [ 3114.259417] Key type ._llcrypt registered [ 3114.260636] Key type .llcrypt registered [ 3114.310523] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3114.702951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3114.734986] alg: No test for adler32 (adler32-zlib) [ 3115.591160] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3115.690030] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 3117.271156] Key type lgssc registered [ 3117.660252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3120.300164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3120.307189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.475686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.837255] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3125.721848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.851237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.261940] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3129.554981] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3131.314341] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.455000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.585621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.428954] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3137.700277] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3145.894569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3161.570862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.571442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3161.574794] Lustre: Skipped 1 previous similar message [ 3161.575055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3165.775785] Lustre: server umount lustre-OST0000 complete [ 3171.808507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.808551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3171.813530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.816524] Lustre: Skipped 2 previous similar messages [ 3173.082658] Lustre: server umount lustre-MDT0000 complete [ 3174.259093] LustreError: 183953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395153 with bad export cookie 3667843455458055684 [ 3174.261483] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.263308] LustreError: 183953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3174.365531] Lustre: server umount lustre-MDT0001 complete [ 3177.696707] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 3178.630390] Key type lgssc unregistered [ 3178.759269] LNet: 187523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3178.761755] LNetError: 187523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3178.768210] LNet: Removed LNI 192.168.202.112@tcp [ 3179.047875] Key type .llcrypt unregistered [ 3179.048801] Key type ._llcrypt unregistered [ 3187.495192] Key type ._llcrypt registered [ 3187.496347] Key type .llcrypt registered [ 3187.539658] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3187.983075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.996298] alg: No test for adler32 (adler32-zlib) [ 3188.854645] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3188.941976] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 3190.519192] Key type lgssc registered [ 3190.877991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.404905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.411329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.579761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.851863] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3198.486073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.604163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.953780] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3202.239621] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3204.008534] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3206.148151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3206.264833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.074739] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3210.277322] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.557268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3224.032055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3224.034264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.038196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.057055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.057528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.062274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.063689] Lustre: Skipped 1 previous similar message [ 3230.178705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.181833] Lustre: Skipped 2 previous similar messages [ 3235.296474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3235.298414] Lustre: Skipped 2 previous similar messages [ 3237.343118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3237.399095] Lustre: server umount lustre-OST0000 complete [ 3239.939238] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ee3515ec000.ost_server_uuid 50 [ 3240.495846] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ee3515ec000.ost_server_uuid in DISCONN state after 0 sec [ 3241.085257] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3255.264042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3255.266617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.271548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3255.273769] Lustre: Skipped 1 previous similar message [ 3257.312546] Lustre: server umount lustre-MDT0000 complete [ 3258.666457] LustreError: 188894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395237 with bad export cookie 3308166590489298646 [ 3258.669327] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3258.669900] LustreError: 188894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3258.807796] Lustre: server umount lustre-MDT0001 complete [ 3261.738220] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:00:40 (1790395240) [ 3264.112152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.303969] LustreError: 192877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.310146] LustreError: 192877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3264.330741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.814490] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3268.860045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.985220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3270.329909] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3272.640077] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3274.343082] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3276.515344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.451155] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3280.702417] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3282.946342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3283.023894] Lustre: lustre-OST0001: new disk, initializing [ 3283.025562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3283.027629] Lustre: 195729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3283.059441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3283.062032] Lustre: Skipped 1 previous similar message [ 3284.419339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3284.423713] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3284.459847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3285.066207] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3288.810792] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3289.582863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3290.079947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.082874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.087427] Lustre: Skipped 1 previous similar message [ 3290.089568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3290.092017] Lustre: Skipped 1 previous similar message [ 3294.694588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.696894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.802649] Lustre: server umount lustre-OST0000 complete [ 3297.705452] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3297.784146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3299.149175] LustreError: 194697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.163267] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3299.245963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3304.268168] LustreError: 194698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.276391] LustreError: 194698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3310.052737] LustreError: 194697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3310.053310] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3310.053441] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.054267] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3310.054271] Lustre: Skipped 3 previous similar messages [ 3310.058372] LustreError: 194697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3310.060724] LustreError: Skipped 1 previous similar message [ 3315.170683] LustreError: 194698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.176865] LustreError: 194698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3315.347637] Lustre: server umount lustre-OST0001 complete [ 3322.720392] Lustre: server umount lustre-MDT0000 complete [ 3323.939245] LustreError: 192857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395303 with bad export cookie 3308166590489299703 [ 3323.942846] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3323.943154] LustreError: 192857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3324.058641] Lustre: server umount lustre-MDT0001 complete [ 3331.619778] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:01:50 (1790395310) [ 3333.703704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.809550] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3333.821453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3333.894101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.158014] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3337.801279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.829582] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3337.841042] Lustre: 199140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3339.234546] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3341.388543] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3343.248694] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3345.632060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.712366] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3347.828411] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3348.620989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3350.412865] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3352.911828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.939341] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3355.049978] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3356.122749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3357.455264] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3359.200140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.200227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.203012] LustreError: Skipped 2 previous similar messages [ 3359.207071] Lustre: Skipped 3 previous similar messages [ 3359.209656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.212339] Lustre: Skipped 6 previous similar messages [ 3364.417593] Lustre: server umount lustre-OST0000 complete [ 3366.371507] LustreError: 200988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.376319] LustreError: 200988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3381.005786] Lustre: server umount lustre-OST0001 complete [ 3388.383192] Lustre: server umount lustre-MDT0000 complete [ 3389.684246] LustreError: 199131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395368 with bad export cookie 3308166590489301089 [ 3389.687037] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3389.688991] LustreError: 199131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3389.832620] Lustre: server umount lustre-MDT0001 complete [ 3397.832584] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:02:57 (1790395377) [ 3403.355134] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 3406.827123] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3419.009278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.094474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3419.103692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3419.139707] Lustre: lustre-MDT0000: new disk, initializing [ 3419.160722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.162797] Lustre: Skipped 3 previous similar messages [ 3419.168220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3420.551961] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3424.725514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.761037] Lustre: 207203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3424.782334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3424.784177] Lustre: Skipped 1 previous similar message [ 3424.821606] Lustre: lustre-MDT0001: new disk, initializing [ 3424.860315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3424.863443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3426.147686] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3429.744275] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3431.484876] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3433.547068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3433.641354] Lustre: lustre-OST0000: new disk, initializing [ 3433.642869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3433.645616] Lustre: 209162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3434.675221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3434.678600] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3434.720132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3435.373304] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3438.841393] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3440.122565] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3440.185503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3441.434991] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3441.495312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3445.218673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.220481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.221155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.221159] Lustre: Skipped 8 previous similar messages [ 3445.224134] Lustre: Skipped 5 previous similar messages [ 3445.231870] LustreError: Skipped 4 previous similar messages [ 3448.208453] Lustre: server umount lustre-OST0000 complete [ 3455.577988] Lustre: server umount lustre-MDT0000 complete [ 3456.868501] LustreError: 207194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395436 with bad export cookie 3308166590489302860 [ 3456.871697] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3456.874553] LustreError: 207194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3457.000691] Lustre: server umount lustre-MDT0001 complete [ 3459.591740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3459.729097] LustreError: 211378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.735612] LustreError: 211378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3461.050879] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3463.841626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3465.221697] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3467.456741] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3469.123827] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3471.123846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3473.060509] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3475.210501] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3476.500170] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3485.707507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3486.921168] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3486.987018] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3493.653769] Lustre: server umount lustre-OST0000 complete [ 3495.690703] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3495.781235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3497.186074] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3520.765393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.891989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3520.894349] Lustre: Skipped 5 previous similar messages [ 3522.805301] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3525.242340] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3525.799050] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3532.810433] Lustre: server umount lustre-OST0000 complete [ 3535.839906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3535.842534] LustreError: Skipped 3 previous similar messages [ 3535.843982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.847691] Lustre: Skipped 4 previous similar messages [ 3540.184818] Lustre: server umount lustre-MDT0000 complete [ 3541.473171] LustreError: 215180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.480542] LustreError: 215180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3541.485842] LustreError: 211405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395520 with bad export cookie 3308166590489303672 [ 3541.490390] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3541.490971] LustreError: 211405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3541.612640] Lustre: server umount lustre-MDT0001 complete [ 3544.513596] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:05:23 (1790395523) [ 3546.741640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.339452] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3551.207071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.887977] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3555.319581] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3557.080867] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3559.266171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.289815] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3563.681112] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3565.236506] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3574.494996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3575.973930] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3576.053275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3578.415989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3578.498784] Lustre: lustre-OST0001: new disk, initializing [ 3578.500713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3578.503208] Lustre: 220847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3580.638127] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3584.512914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3584.515959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3584.554910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3584.570827] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3586.072729] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3586.156780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3587.702550] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3587.801106] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3589.605618] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3589.608265] Lustre: Skipped 13 previous similar messages [ 3590.532144] Lustre: server umount lustre-OST0001 complete [ 3592.663749] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3592.742668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3594.226905] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3594.302748] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3609.933238] LustreError: 219237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.939181] LustreError: 219237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3618.073297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3620.240899] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3622.838694] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3623.337410] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3623.398946] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3623.403422] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3623.403470] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3626.044175] Lustre: server umount lustre-OST0001 complete [ 3644.178826] Lustre: server umount lustre-OST0000 complete [ 3651.550165] Lustre: server umount lustre-MDT0000 complete [ 3652.689919] LustreError: 220845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395631 with bad export cookie 3308166590489304603 [ 3652.692784] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3652.693139] LustreError: 220845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3658.851909] Lustre: server umount lustre-MDT0001 complete [ 3666.002914] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:07:25 (1790395645) [ 3667.993337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.103039] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3668.115383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3668.192580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.194321] Lustre: Skipped 5 previous similar messages [ 3669.403766] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3671.987436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.018030] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3672.027573] Lustre: 226075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3673.342238] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3675.527636] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3677.216806] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3679.217325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3679.285326] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3681.172479] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3683.358123] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3686.905911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.845094] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3691.021742] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3694.536222] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3695.780143] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3695.852389] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3697.079434] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3697.148027] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3698.716265] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ee3454b3000.ost_server_uuid 50 [ 3699.178255] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ee3454b3000.ost_server_uuid in FULL state after 0 sec [ 3699.470755] Lustre: Permanently deactivating lustre-OST0001 [ 3699.475597] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3700.201266] Lustre: Permanently reactivating lustre-OST0001 [ 3701.730109] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.732364] LustreError: Skipped 6 previous similar messages [ 3701.733320] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.736723] Lustre: Skipped 10 previous similar messages [ 3707.199812] Lustre: server umount lustre-OST0001 complete [ 3714.646778] Lustre: server umount lustre-OST0000 complete [ 3730.399098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3730.463257] Lustre: server umount lustre-MDT0000 complete [ 3731.695196] LustreError: 226068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395710 with bad export cookie 3308166590489305877 [ 3731.698643] LustreError: 226068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3731.698986] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3731.808100] Lustre: server umount lustre-MDT0001 complete [ 3739.395401] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:08:38 (1790395718) [ 3741.832577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3741.936454] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3741.947570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3741.951995] Lustre: Skipped 5 previous similar messages [ 3743.386535] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3746.162461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.194614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3746.196661] Lustre: Skipped 1 previous similar message [ 3746.206623] Lustre: 232758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3747.665987] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3750.024352] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3751.936282] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3754.104968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3754.204197] Lustre: Permanently deactivating lustre-OST0000 [ 3754.209447] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3756.212471] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3758.719107] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3760.968934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3762.630650] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3762.634323] LustreError: 235903:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3762.638431] LustreError: 235903:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3762.680962] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3763.023215] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3765.452394] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3766.011344] Lustre: Permanently reactivating lustre-OST0000 [ 3771.898286] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3771.901529] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3771.905498] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3771.908048] Lustre: Skipped 1 previous similar message [ 3771.919069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3771.926657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3780.927425] Lustre: server umount lustre-OST0001 complete [ 3783.139494] LustreError: 234603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.144752] LustreError: 234603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 3788.235522] Lustre: server umount lustre-OST0000 complete [ 3789.640167] Lustre: server umount lustre-MDT0000 complete [ 3790.924323] LustreError: 233498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395770 with bad export cookie 3308166590489307508 [ 3790.928041] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3790.928326] LustreError: 233498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3791.041636] Lustre: server umount lustre-MDT0001 complete [ 3794.647629] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 3795.675386] Key type lgssc unregistered [ 3795.806450] LNet: 237603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3795.810565] LNetError: 237603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3795.822294] LNet: Removed LNI 192.168.202.112@tcp [ 3796.128324] Key type .llcrypt unregistered [ 3796.129916] Key type ._llcrypt unregistered [ 3805.072876] Key type ._llcrypt registered [ 3805.074514] Key type .llcrypt registered [ 3805.116380] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3805.496939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3805.534422] alg: No test for adler32 (adler32-zlib) [ 3806.392573] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 3806.483720] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 3808.063099] Key type lgssc registered [ 3808.393592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3811.706382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3811.712638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3812.860141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.174824] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3816.999350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.034945] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3817.044533] Lustre: 239032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3817.049179] Lustre: Permanently deactivating lustre-MDT0001 [ 3817.055639] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3817.163330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3818.499125] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3820.964609] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3822.059850] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3822.062520] LustreError: 239073:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3822.067469] LustreError: 239073:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3822.072161] LustreError: 239073:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3822.777152] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3824.956155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3825.061213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.934774] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3827.107904] LustreError: 240881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.110496] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3827.115285] LustreError: 240881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3829.262530] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3831.479880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3831.547232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3833.488371] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3835.853757] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3836.903838] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3837.932340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3837.932674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3841.752290] Lustre: Permanently reactivating lustre-MDT0001 [ 3841.758519] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3841.761544] Lustre: Skipped 2 previous similar messages [ 3841.788236] Lustre: lustre-MDT0001: Client 9e97b75a-bf41-47e5-bc2d-f5c5d1d48215 (at 192.168.202.12@tcp) reconnecting [ 3843.532300] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3844.032566] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3845.798203] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3845.883051] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3847.279286] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3847.353023] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3848.180151] Lustre: Permanently deactivating lustre-MDT0001 [ 3848.185415] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3848.187861] Lustre: Skipped 2 previous similar messages [ 3853.319459] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3855.772716] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3855.775545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.780264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.400099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.402619] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.406578] Lustre: Skipped 2 previous similar messages [ 3861.991703] Lustre: server umount lustre-MDT0000 complete [ 3863.307551] LustreError: 239781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395842 with bad export cookie 11155437796987003945 [ 3863.312515] LustreError: 239781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3863.312623] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.439732] Lustre: server umount lustre-MDT0001 complete [ 3870.841624] Lustre: server umount lustre-OST0000 complete [ 3874.518125] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:10:53 (1790395853) [ 3878.043955] Lustre: server umount lustre-OST0001 complete [ 3880.164232] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 3883.512743] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 3895.320275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.414807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3895.418462] Lustre: Skipped 2 previous similar messages [ 3895.427640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3895.471693] Lustre: lustre-MDT0000: new disk, initializing [ 3895.495552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.501610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3896.962289] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3900.877441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.918138] Lustre: 247281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3900.941489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3900.943416] Lustre: Skipped 1 previous similar message [ 3900.975919] Lustre: lustre-MDT0001: new disk, initializing [ 3901.012925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3901.015832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3902.272064] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3905.797833] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3907.414895] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3909.478770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3909.574957] Lustre: lustre-OST0000: new disk, initializing [ 3909.576899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3909.579895] Lustre: 249249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3909.608641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.610270] Lustre: Skipped 1 previous similar message [ 3911.293589] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3911.923669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3911.926661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3911.952783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3914.694163] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3915.291784] LustreError: 247290:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3919.325162] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3919.377931] Lustre: lustre-OST0001: new disk, initializing [ 3919.379335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3919.381372] Lustre: 250363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3921.142080] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 3924.557973] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3925.012193] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3925.383075] LustreError: 247290:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3925.388385] LustreError: 250019:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3928.571672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3928.575299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3928.601255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3935.479053] LustreError: 250019:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3938.788945] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3938.789929] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.791858] LustreError: Skipped 1 previous similar message [ 3938.795504] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3938.798529] Lustre: Skipped 4 previous similar messages [ 3942.265844] Lustre: server umount lustre-OST0001 complete [ 3943.906307] LustreError: 249234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.911744] LustreError: 249234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3949.025602] LustreError: 251240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.029512] LustreError: 251240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3954.145128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.146325] LustreError: 251240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3954.146830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3954.146835] Lustre: Skipped 1 previous similar message [ 3954.147906] Lustre: Skipped 2 previous similar messages [ 3954.151420] LustreError: 251240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3959.264811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3959.265872] LustreError: 251240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.266918] Lustre: Skipped 2 previous similar messages [ 3959.270425] LustreError: 251240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3960.142081] Lustre: server umount lustre-OST0000 complete [ 3962.335802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3962.338101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.583513] Lustre: server umount lustre-MDT0000 complete [ 3968.837369] LustreError: 249248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790395948 with bad export cookie 11155437796987006759 [ 3968.840972] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.844360] LustreError: 249248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3983.327147] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3983.424623] Lustre: server umount lustre-MDT0001 complete [ 3987.155909] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 3988.219670] Key type lgssc unregistered [ 3988.344455] LNet: 252415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3988.347309] LNetError: 252415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3988.358367] LNet: Removed LNI 192.168.202.112@tcp [ 3988.689437] Key type .llcrypt unregistered [ 3988.691034] Key type ._llcrypt unregistered [ 4002.824568] Key type ._llcrypt registered [ 4002.825573] Key type .llcrypt registered [ 4002.865639] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4003.312575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4003.352387] alg: No test for adler32 (adler32-zlib) [ 4004.222879] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4004.324231] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4005.911119] Key type lgssc registered [ 4006.273907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4008.799626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4008.807338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4009.899106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4009.906222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4009.964923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.333955] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4014.194154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4014.226096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4014.236173] Lustre: 254611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4014.327290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4015.734164] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4018.165132] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4020.056814] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4022.389110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4022.457769] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4022.520202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4024.474361] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4026.839798] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4032.530289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4078.564098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.564638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.565736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4078.569393] Lustre: Skipped 1 previous similar message [ 4078.572095] LustreError: Skipped 1 previous similar message [ 4082.775688] Lustre: server umount lustre-OST0000 complete [ 4084.229394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4088.229674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4088.342635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4090.151372] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4092.515846] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4098.593239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.596766] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.596873] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4098.602743] Lustre: Skipped 1 previous similar message [ 4129.250059] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4129.251473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.252254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.252258] Lustre: Skipped 1 previous similar message [ 4129.253755] LustreError: Skipped 1 previous similar message [ 4129.259023] Lustre: Skipped 1 previous similar message [ 4131.797738] Lustre: server umount lustre-OST0000 complete [ 4134.368766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.369494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4134.374125] Lustre: Skipped 1 previous similar message [ 4134.376694] Lustre: Skipped 1 previous similar message [ 4139.162124] Lustre: server umount lustre-MDT0000 complete [ 4140.436095] LustreError: 255351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396119 with bad export cookie 8297082010435716902 [ 4140.439205] LustreError: 255351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4140.439989] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4140.550497] Lustre: server umount lustre-MDT0001 complete [ 4144.254973] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4145.313184] Key type lgssc unregistered [ 4145.447386] LNet: 260043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4145.449501] LNetError: 260043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4145.456307] LNet: Removed LNI 192.168.202.112@tcp [ 4145.794708] Key type .llcrypt unregistered [ 4145.795870] Key type ._llcrypt unregistered [ 4155.150067] Key type ._llcrypt registered [ 4155.151043] Key type .llcrypt registered [ 4155.188826] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4155.519184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4155.527445] alg: No test for adler32 (adler32-zlib) [ 4156.391660] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4156.485521] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4158.071127] Key type lgssc registered [ 4158.437512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4161.029563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4161.034987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.172757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4163.513200] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4166.392156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.520217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.900037] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4170.295501] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4172.060183] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4174.279361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4174.394412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.318377] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4178.734417] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4188.713082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4196.831795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.833728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.837288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.881356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.881917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.882624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4202.895480] Lustre: server umount lustre-OST0000 complete [ 4207.008447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.009317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4207.013610] Lustre: Skipped 1 previous similar message [ 4207.016920] Lustre: Skipped 1 previous similar message [ 4210.262812] Lustre: server umount lustre-MDT0000 complete [ 4211.463428] LustreError: 264713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396190 with bad export cookie 2788955543941819211 [ 4211.467067] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.467646] LustreError: 264713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4211.582878] Lustre: server umount lustre-MDT0001 complete [ 4215.376531] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4216.506283] Key type lgssc unregistered [ 4216.644409] LNet: 265751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4216.647170] LNetError: 265751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4216.655371] LNet: Removed LNI 192.168.202.112@tcp [ 4216.992390] Key type .llcrypt unregistered [ 4216.994054] Key type ._llcrypt unregistered [ 4224.514795] Key type ._llcrypt registered [ 4224.516444] Key type .llcrypt registered [ 4224.559856] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4225.009420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4225.019396] alg: No test for adler32 (adler32-zlib) [ 4225.864335] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4225.944294] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4227.519133] Key type lgssc registered [ 4227.857524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4230.249756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4230.256149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.382669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.602383] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4235.226144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.333511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4236.574983] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4238.770842] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4240.474109] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4242.505834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4242.604786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.278684] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4246.538949] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4255.910874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4261.346278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.347140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4261.348584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4261.350062] Lustre: Skipped 1 previous similar message [ 4261.357395] LustreError: Skipped 1 previous similar message [ 4262.735990] Lustre: server umount lustre-OST0000 complete [ 4265.951908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4265.956065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.962029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4265.963881] Lustre: Skipped 1 previous similar message [ 4266.073038] Lustre: server umount lustre-MDT0000 complete [ 4266.463977] LustreError: 266947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.468725] LustreError: 266947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4267.303241] LustreError: 266927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396246 with bad export cookie 7364010846758135836 [ 4267.306486] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4267.307203] LustreError: 266927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4267.423425] Lustre: server umount lustre-MDT0001 complete [ 4271.219418] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4272.274321] Key type lgssc unregistered [ 4272.408395] LNet: 270638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4272.411287] LNetError: 270638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4272.424400] LNet: Removed LNI 192.168.202.112@tcp [ 4272.739474] Key type .llcrypt unregistered [ 4272.740827] Key type ._llcrypt unregistered [ 4282.209277] Key type ._llcrypt registered [ 4282.210309] Key type .llcrypt registered [ 4282.244154] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4282.591815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4282.635987] alg: No test for adler32 (adler32-zlib) [ 4283.486068] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4283.569795] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4285.151120] Key type lgssc registered [ 4285.525093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4288.057826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4288.065398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.221364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4290.650854] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4293.570906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.707065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4295.074106] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4297.564437] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4299.349815] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4301.651860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.777677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.854743] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4306.393215] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4315.113745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4325.346571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.346609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4325.347708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4325.350119] Lustre: Skipped 1 previous similar message [ 4326.221743] Lustre: server umount lustre-OST0000 complete [ 4329.440025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4329.444050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.451081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4329.454228] Lustre: Skipped 1 previous similar message [ 4329.561709] Lustre: server umount lustre-MDT0000 complete [ 4330.465230] LustreError: 274514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.466375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.469577] LustreError: 274514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4330.785254] LustreError: 272006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396310 with bad export cookie 11130764059399339302 [ 4330.791267] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4330.792526] LustreError: 272006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4330.917753] Lustre: server umount lustre-MDT0001 complete [ 4334.587663] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4335.572609] Key type lgssc unregistered [ 4335.697397] LNet: 276349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4335.699613] LNetError: 276349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4335.709317] LNet: Removed LNI 192.168.202.112@tcp [ 4336.008966] Key type .llcrypt unregistered [ 4336.010476] Key type ._llcrypt unregistered [ 4343.083485] Key type ._llcrypt registered [ 4343.084394] Key type .llcrypt registered [ 4343.125985] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4343.486719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4343.496094] alg: No test for adler32 (adler32-zlib) [ 4344.349061] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4344.438634] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4346.015084] Key type lgssc registered [ 4346.365338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4348.917432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4348.922861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.072567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.357644] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4354.331022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.461616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4355.947891] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4358.326733] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4360.211970] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4362.490163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4362.605961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4364.512593] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4366.969408] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4374.886404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4385.248525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4385.248968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4385.252220] LustreError: Skipped 1 previous similar message [ 4385.258958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4385.262376] Lustre: Skipped 1 previous similar message [ 4387.020520] Lustre: server umount lustre-OST0000 complete [ 4390.369164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4390.369389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4390.369966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4390.379911] Lustre: Skipped 2 previous similar messages [ 4394.387986] Lustre: server umount lustre-MDT0000 complete [ 4395.488050] LustreError: 277565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.492417] LustreError: 277565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4395.608183] LustreError: 277526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396374 with bad export cookie 17849576808535413109 [ 4395.611843] LustreError: 277526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4395.612654] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.728139] Lustre: server umount lustre-MDT0001 complete [ 4399.197478] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4400.185933] Key type lgssc unregistered [ 4400.301319] LNet: 281272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4400.303826] LNetError: 281272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4400.312291] LNet: Removed LNI 192.168.202.112@tcp [ 4400.609544] Key type .llcrypt unregistered [ 4400.611202] Key type ._llcrypt unregistered [ 4404.121151] Key type ._llcrypt registered [ 4404.122217] Key type .llcrypt registered [ 4404.158520] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4409.697273] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 4415.877879] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4416.133527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4416.140852] alg: No test for adler32 (adler32-zlib) [ 4416.990277] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4417.079884] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4418.655122] Key type lgssc registered [ 4419.024042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4430.085711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4430.093677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.174612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4431.186154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4431.226246] Lustre: lustre-MDT0000: new disk, initializing [ 4431.247258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.253186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4432.675315] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4436.775044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.825047] Lustre: 284790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4436.849390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4436.850917] Lustre: Skipped 1 previous similar message [ 4436.886964] Lustre: lustre-MDT0001: new disk, initializing [ 4436.918810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4436.930606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4436.936177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4438.275273] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4441.926634] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4443.703181] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4446.013397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.110865] Lustre: lustre-OST0000: new disk, initializing [ 4446.113253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4446.116813] Lustre: 286759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4446.143969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.967748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4446.972968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4447.021365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4448.071897] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4451.866105] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4453.309793] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4453.380665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4454.776733] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4454.848249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4457.441766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4457.443860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.446853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4457.446858] Lustre: Skipped 1 previous similar message [ 4457.556162] Lustre: server umount lustre-OST0000 complete [ 4462.561561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4462.562111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.563237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4462.563243] Lustre: Skipped 1 previous similar message [ 4462.563949] LustreError: Skipped 1 previous similar message [ 4462.568204] Lustre: Skipped 1 previous similar message [ 4467.680528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.683940] Lustre: Skipped 2 previous similar messages [ 4472.800322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.802814] Lustre: Skipped 1 previous similar message [ 4473.311120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4473.379303] Lustre: server umount lustre-MDT0000 complete [ 4474.701260] LustreError: 284784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396453 with bad export cookie 17166897414394377212 [ 4474.704415] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4474.705243] LustreError: 284784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4474.830841] Lustre: server umount lustre-MDT0001 complete [ 4477.700292] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:20:56 (1790396456) [ 4480.067807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.268163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.616420] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4484.472450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.602354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4486.063881] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4488.557730] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4490.347972] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4492.573530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.695283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4494.632556] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4496.995842] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4511.202106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4511.202701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.203610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4511.210154] Lustre: Skipped 1 previous similar message [ 4516.311254] Lustre: server umount lustre-OST0000 complete [ 4520.415837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4520.419091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.424217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4520.426850] Lustre: Skipped 1 previous similar message [ 4523.731339] Lustre: server umount lustre-MDT0000 complete [ 4524.946592] LustreError: 289147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396504 with bad export cookie 17166897414394378073 [ 4524.951796] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.081884] Lustre: server umount lustre-MDT0001 complete [ 4528.555602] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4529.537195] Key type lgssc unregistered [ 4529.661362] LNet: 292717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4529.663720] LNetError: 292717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4529.674288] LNet: Removed LNI 192.168.202.112@tcp [ 4529.963175] Key type .llcrypt unregistered [ 4529.964217] Key type ._llcrypt unregistered [ 4538.263068] Key type ._llcrypt registered [ 4538.264541] Key type .llcrypt registered [ 4538.302749] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 4544.070901] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4544.480295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4544.488802] alg: No test for adler32 (adler32-zlib) [ 4545.339602] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4545.429143] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4547.007174] Key type lgssc registered [ 4547.352683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4561.717250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4561.722356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.802456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4562.812046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4562.850444] Lustre: lustre-MDT0000: new disk, initializing [ 4562.870020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.875756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4564.279637] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4568.322036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.369052] Lustre: 296509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4568.393919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4568.396037] Lustre: Skipped 1 previous similar message [ 4568.431146] Lustre: lustre-MDT0001: new disk, initializing [ 4568.464468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4568.476886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4568.480143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4569.816599] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4573.493894] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4575.336277] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4577.523497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4577.621278] Lustre: lustre-OST2710: new disk, initializing [ 4577.623240] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4577.625611] Lustre: 298480:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4577.652994] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4579.571322] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4583.308212] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4585.568049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.634803] Lustre: lustre-OST03e8: new disk, initializing [ 4585.636589] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4585.638955] Lustre: 299532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4585.673029] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4587.210099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4587.213396] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4587.216037] Lustre: Skipped 1 previous similar message [ 4587.266734] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4587.271906] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4587.635252] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4589.950396] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.525895] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4592.594765] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4594.037314] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4594.120737] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4595.484925] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4595.565758] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4596.909615] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4596.986738] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4599.263853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4599.266029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.269866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4602.850902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4602.851061] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.853824] Lustre: Skipped 1 previous similar message [ 4602.857340] Lustre: Skipped 2 previous similar messages [ 4604.124172] Lustre: server umount lustre-MDT0000 complete [ 4605.320167] LustreError: 296503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396584 with bad export cookie 4121661982311152589 [ 4605.324222] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.324403] LustreError: 296503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4605.441601] Lustre: server umount lustre-MDT0001 complete [ 4616.199286] Lustre: server umount lustre-OST2710 complete [ 4627.673555] Lustre: server umount lustre-OST03e8 complete [ 4633.419827] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 4636.464136] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4647.856737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.938391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4647.948454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4647.984612] Lustre: lustre-MDT0000: new disk, initializing [ 4648.004492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4648.009974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4648.012358] Lustre: Skipped 1 previous similar message [ 4649.313517] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4653.433489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.484850] Lustre: 304777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.591523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.596325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4654.832275] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4658.429463] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4660.291671] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.503232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.630698] Lustre: lustre-OST0000: new disk, initializing [ 4662.632073] Lustre: Skipped 1 previous similar message [ 4662.633514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4662.635159] Lustre: Skipped 2 previous similar messages [ 4662.637208] Lustre: 306744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4662.664475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.666377] Lustre: Skipped 1 previous similar message [ 4664.535528] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4666.551590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4666.556894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4666.587918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4668.285848] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4669.727873] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4669.819219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4671.194328] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4671.267953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4671.970378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4671.970996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4671.972094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4671.974022] Lustre: Skipped 1 previous similar message [ 4671.977268] LustreError: Skipped 1 previous similar message [ 4677.090253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4677.094022] Lustre: Skipped 2 previous similar messages [ 4682.210234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4682.211966] Lustre: Skipped 1 previous similar message [ 4686.303124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4686.365537] Lustre: server umount lustre-OST0000 complete [ 4689.376041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4689.378827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.448570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4692.450677] Lustre: Skipped 2 previous similar messages [ 4693.659189] Lustre: server umount lustre-MDT0000 complete [ 4694.960128] LustreError: 304813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396674 with bad export cookie 4121661982311154157 [ 4694.963322] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4694.963423] LustreError: 304813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4695.089817] Lustre: server umount lustre-MDT0001 complete [ 4697.843113] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:24:37 (1790396677) [ 4698.353230] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4698.983980] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:24:38 (1790396678) [ 4700.068356] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4708.891473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.973624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4708.980887] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4709.041844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4710.416909] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4713.284850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.313062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4713.321866] Lustre: 310596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4714.946715] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4731.492600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.607957] Lustre: Denying initial registration attempt from nid 192.168.202.112@tcp, specified as failover [ 4731.617866] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4731.631810] LustreError: 312045:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 4731.638641] LustreError: 312045:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4731.646101] LustreError: 312045:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4731.712767] Lustre: server umount lustre-OST0000 complete [ 4731.715802] LustreError: 312045:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4736.480932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4736.482656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.489507] Lustre: Skipped 2 previous similar messages [ 4749.279416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4749.418238] Lustre: server umount lustre-MDT0000 complete [ 4751.651388] LustreError: 310587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396730 with bad export cookie 4121661982311155025 [ 4751.652496] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.656887] LustreError: 310587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4751.816613] Lustre: server umount lustre-MDT0001 complete [ 4759.855433] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4761.639261] Key type lgssc unregistered [ 4761.891457] LNet: 313165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4761.897974] LNetError: 313165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4761.926630] LNet: Removed LNI 192.168.202.112@tcp [ 4762.598931] Key type .llcrypt unregistered [ 4762.601509] Key type ._llcrypt unregistered [ 4768.328795] Key type ._llcrypt registered [ 4768.329864] Key type .llcrypt registered [ 4768.367540] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4768.797615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4768.804841] alg: No test for adler32 (adler32-zlib) [ 4769.687684] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4769.782223] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4771.367176] Key type lgssc registered [ 4771.829254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4798.454857] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4802.249190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4802.255931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.383496] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4803.394384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4803.488734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4805.291537] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4808.839291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.888327] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4808.901403] Lustre: 315902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4809.022952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4810.735139] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4813.952714] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4816.316228] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4832.231876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4832.317196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4832.411857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.514957] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4837.397406] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4842.976640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4842.978809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.979578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4842.983298] Lustre: Skipped 1 previous similar message [ 4842.986835] Lustre: Skipped 1 previous similar message [ 4844.309040] Lustre: server umount lustre-OST0000 complete [ 4848.097664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.100784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4848.103409] Lustre: Skipped 1 previous similar message [ 4853.216824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.219300] Lustre: Skipped 2 previous similar messages [ 4858.335978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4858.338046] Lustre: Skipped 1 previous similar message [ 4860.383112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.450449] Lustre: server umount lustre-MDT0000 complete [ 4861.952726] LustreError: 315894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396841 with bad export cookie 10527094794551188532 [ 4861.955758] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.958290] LustreError: 315894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4862.102740] Lustre: server umount lustre-MDT0001 complete [ 4866.453366] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4867.691553] Key type lgssc unregistered [ 4867.828424] LNet: 319471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4867.832085] LNetError: 319471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4867.841314] LNet: Removed LNI 192.168.202.112@tcp [ 4868.209199] Key type .llcrypt unregistered [ 4868.213394] Key type ._llcrypt unregistered [ 4879.130328] Key type ._llcrypt registered [ 4879.131870] Key type .llcrypt registered [ 4879.180801] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 4879.636576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4879.705321] alg: No test for adler32 (adler32-zlib) [ 4880.600507] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 4880.734866] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 4882.327210] Key type lgssc registered [ 4882.749045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4885.674147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4885.681515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.866105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4888.565833] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4892.088824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4892.237064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4893.995385] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4897.135497] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4899.331242] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4904.101393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.265894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.635435] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4909.631166] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4926.293539] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) [ 4927.970057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.970632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.971748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4927.975854] Lustre: Skipped 2 previous similar messages [ 4927.982887] Lustre: Skipped 2 previous similar messages [ 4930.483192] Lustre: server umount lustre-MDT0000 complete [ 4931.407923] LustreError: 320885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.413678] LustreError: 320885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4932.059491] LustreError: 320887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396911 with bad export cookie 6865926031385200545 [ 4932.062301] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4932.064468] LustreError: 320887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4933.087970] LustreError: 320857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.088600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.102638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4936.527465] LustreError: 320855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.536383] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [ 4941.644050] LustreError: 320856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.644069] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [ 4941.652459] LustreError: 320856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4941.654772] Lustre: Skipped 1 previous similar message [ 4946.399133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4946.483595] Lustre: server umount lustre-MDT0001 complete [ 4949.005832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4952.349832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.666408] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5f48ab592ad2a370 [ 4957.674436] Lustre: MGC192.168.202.112@tcp: Connection restored to 0@lo (at 0@lo) [ 4957.792636] LustreError: 322690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4957.801590] LustreError: 322690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4957.824518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.849626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4959.469139] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4962.782440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.939622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4964.503115] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 4967.413176] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4972.515993] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4973.148502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4975.272339] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4975.942631] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4977.634182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.636733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4977.640571] Lustre: Skipped 1 previous similar message [ 4977.645430] Lustre: Skipped 2 previous similar messages [ 4983.704651] Lustre: server umount lustre-OST0000 complete [ 4987.873428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.878923] Lustre: Skipped 1 previous similar message [ 4991.255800] Lustre: server umount lustre-MDT0000 complete [ 4992.711464] LustreError: 320841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790396971 with bad export cookie 6865926031385207664 [ 4992.713796] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.718155] LustreError: 320841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4992.847546] Lustre: server umount lustre-MDT0001 complete [ 4996.918791] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 4998.149680] Key type lgssc unregistered [ 4998.303988] LNet: 327140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4998.307611] LNetError: 327140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4998.317395] LNet: Removed LNI 192.168.202.112@tcp [ 4998.679912] Key type .llcrypt unregistered [ 4998.681471] Key type ._llcrypt unregistered [ 5009.540157] Key type ._llcrypt registered [ 5009.541459] Key type .llcrypt registered [ 5009.593717] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5009.917126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5009.925049] alg: No test for adler32 (adler32-zlib) [ 5010.820878] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5010.946571] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5012.543218] Key type lgssc registered [ 5013.028174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5016.680482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5016.695390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.896252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.653349] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5023.357056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5023.529240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.514561] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5028.759101] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5031.307161] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5033.761196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.763128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5033.773723] Lustre: Skipped 1 previous similar message [ 5038.882627] Lustre: server umount lustre-MDT0000 complete [ 5039.075137] LustreError: 328529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.089872] LustreError: 328529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5040.553307] LustreError: 328511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397019 with bad export cookie 18265125764055474634 [ 5040.557581] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5040.557689] LustreError: 328511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5040.723666] Lustre: server umount lustre-MDT0001 complete [ 5047.168824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.286901] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5047.300345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5047.391180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.943441] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5052.291941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.353592] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5052.376350] Lustre: 331115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5052.501516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5054.181516] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5057.317820] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5060.458621] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5066.340421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5066.443366] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5066.448465] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5066.452327] LustreError: 332996:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5066.455538] LustreError: 332996:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5066.458298] LustreError: 332996:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5066.490243] Lustre: server umount lustre-OST0000 complete [ 5066.492516] LustreError: 332996:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5072.359890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.482377] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5072.613750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.390333] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5079.278595] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5080.162164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5083.247291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5083.303810] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5083.351255] Lustre: lustre-OST0001: new disk, initializing [ 5083.353537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5083.356458] Lustre: 334538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5083.403716] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.824204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5085.829927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5085.898236] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5086.373448] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5089.858301] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5091.296854] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5091.300616] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.307095] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5091.309481] Lustre: Skipped 2 previous similar messages [ 5096.422329] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5096.426289] Lustre: Skipped 1 previous similar message [ 5097.299882] Lustre: server umount lustre-OST0001 complete [ 5100.512299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5100.517170] LustreError: Skipped 1 previous similar message [ 5100.519360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5100.526088] Lustre: Skipped 1 previous similar message [ 5100.530700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5101.540094] LustreError: 333464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.547504] LustreError: 333464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5106.659678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.660406] LustreError: 333464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5106.663826] Lustre: Skipped 3 previous similar messages [ 5106.669561] LustreError: 333464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5111.779954] LustreError: 333464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.789567] LustreError: 333464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5113.311122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5113.402260] Lustre: server umount lustre-OST0000 complete [ 5116.896792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.897375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5116.902328] Lustre: Skipped 2 previous similar messages [ 5116.904603] Lustre: Skipped 2 previous similar messages [ 5121.119164] Lustre: server umount lustre-MDT0000 complete [ 5122.016093] LustreError: 331121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.020758] LustreError: 331121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5122.484368] LustreError: 331107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397101 with bad export cookie 18265125764055475173 [ 5122.488110] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.489106] LustreError: 331107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5122.626552] Lustre: server umount lustre-MDT0001 complete [ 5126.578724] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 5127.700765] Key type lgssc unregistered [ 5127.831413] LNet: 336492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5127.834376] LNetError: 336492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5127.842480] LNet: Removed LNI 192.168.202.112@tcp [ 5128.177097] Key type .llcrypt unregistered [ 5128.178574] Key type ._llcrypt unregistered [ 5161.168137] Key type ._llcrypt registered [ 5161.169213] Key type .llcrypt registered [ 5161.215305] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 5167.981957] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5168.340393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5168.397041] alg: No test for adler32 (adler32-zlib) [ 5169.267223] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5169.368516] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5170.951223] Key type lgssc registered [ 5171.374478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5183.616759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5183.623089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.706603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5184.716062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5184.751985] Lustre: lustre-MDT0000: new disk, initializing [ 5184.772388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.778339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5186.257882] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5190.652077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5190.705149] Lustre: 341392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5190.734881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5190.737335] Lustre: Skipped 1 previous similar message [ 5190.773687] Lustre: lustre-MDT0001: new disk, initializing [ 5190.806159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5190.819466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5190.822529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5192.352291] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5196.618838] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.901762] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5201.634135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5201.744120] Lustre: lustre-OST0000: new disk, initializing [ 5201.746390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5201.749883] Lustre: 343359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5201.784516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5204.086405] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5207.539970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5207.543132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5207.580603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5207.987650] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5209.543647] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5209.627912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5211.248479] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5211.337230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5212.640558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5212.641632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.644054] LustreError: Skipped 1 previous similar message [ 5212.650994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5212.653645] Lustre: Skipped 1 previous similar message [ 5214.115274] Lustre: server umount lustre-OST0000 complete [ 5216.736311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5216.738778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.743285] Lustre: Skipped 1 previous similar message [ 5216.748094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.762203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.763100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.754853] Lustre: server umount lustre-MDT0000 complete [ 5223.308492] LustreError: 341384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397202 with bad export cookie 337959586493173936 [ 5223.310692] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.316700] LustreError: 341384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5223.431927] Lustre: server umount lustre-MDT0001 complete [ 5226.663870] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:33:25 (1790397205) [ 5229.871774] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:33:28 (1790397208) [ 5233.441839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.638284] LustreError: 346106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5233.649714] LustreError: 346106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5233.676568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5235.272721] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5238.648633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.751738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5238.754493] Lustre: Skipped 1 previous similar message [ 5238.792212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5240.430887] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5243.508182] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5245.626756] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5248.160478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5248.333159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.730252] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5253.577246] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5263.456868] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5264.106423] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5264.730222] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5265.387871] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5266.108056] Lustre: Failing over lustre-MDT0000 [ 5266.299907] Lustre: server umount lustre-MDT0000 complete [ 5268.960674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5268.962484] LustreError: 346103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.965205] Lustre: Skipped 2 previous similar messages [ 5268.971141] LustreError: 346103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5270.354269] LustreError: 346103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.080177] LustreError: 346107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.085772] LustreError: 346107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5279.200731] LustreError: 346103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.208126] LustreError: 346103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5279.522486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.573302] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5279.686530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5279.716739] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5280.588603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5281.242157] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5284.837345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5284.848969] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5284.868924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5287.071216] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5287.667166] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5289.783317] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5290.590473] Lustre: Failing over lustre-MDT0000 [ 5290.721728] Lustre: server umount lustre-MDT0000 complete [ 5290.838455] LustreError: 346101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.074523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5295.079088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5295.081382] Lustre: Skipped 2 previous similar messages [ 5304.372285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.453042] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5304.576309] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5306.194833] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5306.291828] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5309.924922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5309.928340] Lustre: Skipped 2 previous similar messages [ 5309.940592] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5309.960166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5312.248187] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5312.874341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5320.162298] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.162774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5320.163180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.172977] Lustre: Skipped 1 previous similar message [ 5321.642135] Lustre: server umount lustre-MDT0000 complete [ 5323.277103] LustreError: 347946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397302 with bad export cookie 337959586493176617 [ 5323.280277] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5323.282211] LustreError: 347946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5323.453807] Lustre: server umount lustre-MDT0001 complete [ 5335.471897] Lustre: server umount lustre-OST0000 complete [ 5341.030182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.198365] LustreError: 352436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.206155] LustreError: 352436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5341.225175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.227700] Lustre: Skipped 1 previous similar message [ 5342.743147] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5346.156433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.954645] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5350.795732] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5353.002581] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5355.622648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5357.926121] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5360.789160] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5367.014352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5377.508634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.509160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5377.510165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5377.510170] Lustre: Skipped 2 previous similar messages [ 5377.513261] Lustre: Skipped 2 previous similar messages [ 5377.519516] LustreError: Skipped 1 previous similar message [ 5378.734379] Lustre: server umount lustre-OST0000 complete [ 5386.331430] Lustre: server umount lustre-MDT0000 complete [ 5387.744134] LustreError: 352455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.750823] LustreError: 352455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5387.820195] LustreError: 354277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397367 with bad export cookie 337959586493177338 [ 5387.822499] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.824367] LustreError: 354277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5387.968398] Lustre: server umount lustre-MDT0001 complete [ 5391.972523] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 5393.104400] Key type lgssc unregistered [ 5393.257518] LNet: 355985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5393.260294] LNetError: 355985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5393.269324] LNet: Removed LNI 192.168.202.112@tcp [ 5393.624792] Key type .llcrypt unregistered [ 5393.626095] Key type ._llcrypt unregistered [ 5404.409384] Key type ._llcrypt registered [ 5404.411210] Key type .llcrypt registered [ 5404.450310] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5404.800445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5404.813152] alg: No test for adler32 (adler32-zlib) [ 5405.689175] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5405.787659] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5407.375408] Key type lgssc registered [ 5407.841653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5411.005886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5411.018420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.199478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5413.667249] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5417.246767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5417.458453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5419.355503] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5422.237206] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5424.556251] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5427.085714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.216309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.392027] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5432.164038] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5437.478820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5443.039831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5443.043066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.047948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.650652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.651395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.660741] Lustre: Skipped 2 previous similar messages [ 5448.672429] Lustre: server umount lustre-MDT0000 complete [ 5450.113540] LustreError: 359979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397429 with bad export cookie 3434797045819661253 [ 5450.118090] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5450.118185] LustreError: 359979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5450.232813] Lustre: server umount lustre-MDT0001 complete [ 5460.902613] Lustre: server umount lustre-OST0000 complete [ 5465.949579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.124382] LustreError: 361196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5466.132122] LustreError: 361196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5466.161120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.729146] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5470.923630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.082670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5472.644801] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5475.670389] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5477.707327] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5480.242809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.396800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.574268] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5485.345898] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5491.690714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5507.041428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.042085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5507.042600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.046162] Lustre: Skipped 2 previous similar messages [ 5509.272576] Lustre: server umount lustre-MDT0000 complete [ 5510.639094] LustreError: 361178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397489 with bad export cookie 3434797045819662352 [ 5510.641991] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5510.644714] LustreError: 361178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5510.752698] Lustre: server umount lustre-MDT0001 complete [ 5522.469502] Lustre: server umount lustre-OST0000 complete [ 5527.768427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.921156] LustreError: 365123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5527.944145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.323500] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5532.164149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.702093] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5536.184769] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5538.297204] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5540.614984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.788614] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5545.322144] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5554.087040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5559.267406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5559.267429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5559.273351] Lustre: Skipped 1 previous similar message [ 5559.275158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5559.277405] Lustre: Skipped 3 previous similar messages [ 5563.406715] Lustre: server umount lustre-OST0000 complete [ 5568.479829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5568.482861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.487845] Lustre: Skipped 1 previous similar message [ 5568.489678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5570.779465] Lustre: server umount lustre-MDT0000 complete [ 5572.146579] LustreError: 365105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397551 with bad export cookie 3434797045819663360 [ 5572.148058] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.150947] LustreError: 365105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5572.270516] Lustre: server umount lustre-MDT0001 complete [ 5576.139902] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 5577.207316] Key type lgssc unregistered [ 5577.338427] LNet: 368673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5577.341768] LNetError: 368673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5577.353331] LNet: Removed LNI 192.168.202.112@tcp [ 5577.721705] Key type .llcrypt unregistered [ 5577.723685] Key type ._llcrypt unregistered [ 5587.366676] Key type ._llcrypt registered [ 5587.368340] Key type .llcrypt registered [ 5587.409534] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5587.712568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5587.724320] alg: No test for adler32 (adler32-zlib) [ 5588.577241] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5588.662572] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5590.247178] Key type lgssc registered [ 5590.577359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5593.037673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5593.044136] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.046721] LustreError: 370087:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5593.054080] LustreError: 370087:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5593.056102] LustreError: 370087:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5593.058056] LustreError: 370087:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5593.060568] LustreError: 370087:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5595.405596] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5595.407954] LustreError: 370484:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5595.416954] LustreError: 370484:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5595.419075] LustreError: 370484:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5595.421123] LustreError: 370484:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5595.423540] LustreError: 370484:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5601.699663] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 5602.779800] Key type lgssc unregistered [ 5602.920697] LNet: 371486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5602.924248] LNetError: 371486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5602.936277] LNet: Removed LNI 192.168.202.112@tcp [ 5603.245860] Key type .llcrypt unregistered [ 5603.247080] Key type ._llcrypt unregistered [ 5609.225055] Key type ._llcrypt registered [ 5609.226118] Key type .llcrypt registered [ 5609.266521] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 5615.486125] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5615.884897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.897834] alg: No test for adler32 (adler32-zlib) [ 5616.767434] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5616.855456] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5618.439157] Key type lgssc registered [ 5618.855790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5630.225062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5630.231068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.320492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5631.335050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5631.375087] Lustre: lustre-MDT0000: new disk, initializing [ 5631.402765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.408976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5632.787779] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5636.784923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.825079] Lustre: 374613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5636.849218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5636.851417] Lustre: Skipped 1 previous similar message [ 5636.892365] Lustre: lustre-MDT0001: new disk, initializing [ 5636.922875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5636.935819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5636.939339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5638.187938] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5641.885070] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5643.760376] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.948477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5646.047923] Lustre: lustre-OST0000: new disk, initializing [ 5646.049875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5646.052032] Lustre: 376580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5646.085539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.063728] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5651.817316] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5653.208618] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5653.496795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5653.500689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5653.532788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5654.302218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5655.748981] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5655.831981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5658.596178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5658.596375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.599169] LustreError: Skipped 1 previous similar message [ 5658.604851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5658.608592] Lustre: Skipped 1 previous similar message [ 5662.552485] Lustre: server umount lustre-OST0000 complete [ 5667.807903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5667.811152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.815441] Lustre: Skipped 1 previous similar message [ 5667.817301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5668.832137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5668.832770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.974724] Lustre: server umount lustre-MDT0000 complete [ 5671.238221] LustreError: 376579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397650 with bad export cookie 17167399732256995991 [ 5671.240834] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5671.242140] LustreError: 376579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5671.351797] Lustre: server umount lustre-MDT0001 complete [ 5674.002910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5674.535454] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:40:53 (1790397653) [ 5676.623483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.785853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.041816] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5680.654670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.157834] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5684.608399] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5686.423737] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5688.607048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5688.714308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5688.716388] Lustre: Skipped 1 previous similar message [ 5690.636530] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5693.138248] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5695.545423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5695.615726] Lustre: lustre-OST0001: new disk, initializing [ 5695.617786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5695.621135] Lustre: 381880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5697.697809] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5699.653790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5699.658457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5699.693741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5701.568282] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5704.672059] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5704.674729] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.678846] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5704.681054] Lustre: Skipped 1 previous similar message [ 5709.503837] Lustre: server umount lustre-OST0001 complete [ 5712.356078] LustreError: 380864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.362712] LustreError: 380864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5717.475794] LustreError: 380864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.482061] LustreError: 380864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5722.594027] LustreError: 381248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.594885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.595610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5722.595615] Lustre: Skipped 4 previous similar messages [ 5722.601437] LustreError: 381248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5722.612843] Lustre: Skipped 2 previous similar messages [ 5722.614697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5722.617899] LustreError: Skipped 1 previous similar message [ 5727.714535] LustreError: 382801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.720664] LustreError: 382801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5727.821968] Lustre: server umount lustre-OST0000 complete [ 5732.319786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5732.322982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.328239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5732.330810] Lustre: Skipped 3 previous similar messages [ 5735.266676] Lustre: server umount lustre-MDT0000 complete [ 5736.588172] LustreError: 379070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397715 with bad export cookie 17167399732256996859 [ 5736.591436] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.592319] LustreError: 379070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5742.767580] Lustre: server umount lustre-MDT0001 complete [ 5746.400098] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 5747.439739] Key type lgssc unregistered [ 5747.571332] LNet: 383861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5747.574300] LNetError: 383861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5747.584328] LNet: Removed LNI 192.168.202.112@tcp [ 5747.914914] Key type .llcrypt unregistered [ 5747.915991] Key type ._llcrypt unregistered [ 5766.399478] Key type ._llcrypt registered [ 5766.400363] Key type .llcrypt registered [ 5766.434922] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5766.836479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5766.843719] alg: No test for adler32 (adler32-zlib) [ 5767.690276] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5767.770214] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5769.351149] Key type lgssc registered [ 5769.714457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5772.409784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5772.415516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.496448] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5773.503079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5773.555901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5774.898946] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5777.763813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.801556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5777.811964] Lustre: 386629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5777.912850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5779.345542] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5781.829300] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5783.636654] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5787.616723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.617425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.620636] Lustre: Skipped 1 previous similar message [ 5790.487076] Lustre: server umount lustre-MDT0000 complete [ 5791.767329] LustreError: 387370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397771 with bad export cookie 395582687960113473 [ 5791.770152] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5791.771931] LustreError: 387370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5791.889767] Lustre: server umount lustre-MDT0001 complete [ 5794.810592] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:42:54 (1790397774) [ 5796.943977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.111556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5798.434402] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5801.260308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.371664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5802.725418] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5805.075467] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5806.837637] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.983450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.045514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5809.105489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5810.923671] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5813.250881] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5817.869796] Lustre: Permanently deactivating lustre-OST0000 [ 5817.875229] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5819.631397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5820.988222] LustreError: 392027:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5820.991928] LustreError: 392027:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5820.994276] LustreError: 392027:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5821.565081] LustreError: 392075:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5821.567420] LustreError: 392075:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5828.307298] Lustre: server umount lustre-OST0000 complete [ 5829.149636] LustreError: 392325:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5829.151915] LustreError: 392325:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5832.159719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5832.162299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5832.166326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.167845] Lustre: Skipped 1 previous similar message [ 5834.720176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5834.720599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.840164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5839.842435] Lustre: Skipped 2 previous similar messages [ 5839.865642] Lustre: server umount lustre-MDT0000 complete [ 5841.081123] LustreError: 390979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397820 with bad export cookie 395582687960114040 [ 5841.084430] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5841.084489] LustreError: 390979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5851.658975] Lustre: server umount lustre-MDT0001 complete [ 5854.427518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.690339] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5856.516784] LustreError: 393484:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5858.379140] Lustre: 393822:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.112@tcp [ 5858.994175] Lustre: server umount MGS complete [ 5862.355525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.500920] LustreError: 394545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.506240] LustreError: 394545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5862.506261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5862.509994] Lustre: Skipped 1 previous similar message [ 5862.529559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5863.849127] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5866.500508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.591109] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5867.830320] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5870.054110] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5871.829479] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5872.421335] Lustre: Permanently reactivating lustre-OST0000 [ 5872.427173] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5872.429545] Lustre: Skipped 2 previous similar messages [ 5893.599122] LustreError: 396179:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5896.458240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.555889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.558530] Lustre: Skipped 1 previous similar message [ 5898.356797] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5900.553863] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5913.058276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.058624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5913.061782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5913.063867] LustreError: Skipped 1 previous similar message [ 5913.067131] Lustre: Skipped 1 previous similar message [ 5914.187826] Lustre: server umount lustre-OST0000 complete [ 5917.531427] Lustre: server umount lustre-MDT0000 complete [ 5918.176168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5918.176313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.181234] Lustre: Skipped 1 previous similar message [ 5918.182959] LustreError: 394545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.187190] LustreError: 394545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5918.626225] LustreError: 394527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397897 with bad export cookie 395582687960115062 [ 5918.629078] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.629244] LustreError: 394527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5918.727408] Lustre: server umount lustre-MDT0001 complete [ 5921.964438] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 5922.928598] Key type lgssc unregistered [ 5923.042348] LNet: 399068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5923.044943] LNetError: 399068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5923.052303] LNet: Removed LNI 192.168.202.112@tcp [ 5923.340789] Key type .llcrypt unregistered [ 5923.342413] Key type ._llcrypt unregistered [ 5929.045057] Key type ._llcrypt registered [ 5929.046723] Key type .llcrypt registered [ 5929.082917] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 5934.919413] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 5935.265402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5935.281957] alg: No test for adler32 (adler32-zlib) [ 5936.118868] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 5936.198643] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 5937.775188] Key type lgssc registered [ 5938.089041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5948.685488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5948.690793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.771825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5949.780933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5949.817571] Lustre: lustre-MDT0000: new disk, initializing [ 5949.837238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.843963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5951.145104] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5955.169846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5955.208097] Lustre: 402192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5955.230882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5955.233103] Lustre: Skipped 1 previous similar message [ 5955.267255] Lustre: lustre-MDT0001: new disk, initializing [ 5955.298418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5955.309311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5955.312642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5956.591337] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5960.187668] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5961.900876] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5963.966211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.083625] Lustre: lustre-OST0000: new disk, initializing [ 5964.085417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5964.088289] Lustre: 404157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5964.119266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.033360] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 5966.775540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5966.779670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5966.818043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5969.589430] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5970.981128] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5971.056045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5972.384395] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5972.456260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5977.057391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.058545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5977.060121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.061307] Lustre: Skipped 1 previous similar message [ 5977.063196] LustreError: Skipped 1 previous similar message [ 5982.177626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5982.180647] Lustre: Skipped 2 previous similar messages [ 5987.295121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5987.298599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.300689] Lustre: Skipped 1 previous similar message [ 5987.360531] Lustre: server umount lustre-OST0000 complete [ 5991.391775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5991.394191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.398197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.416941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5994.773582] Lustre: server umount lustre-MDT0000 complete [ 5996.010092] LustreError: 402184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790397975 with bad export cookie 527359961368406729 [ 5996.012804] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.014262] LustreError: 402184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5996.127815] Lustre: server umount lustre-MDT0001 complete [ 5998.824504] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5999.328213] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:46:18 (1790397978) [ 6001.454167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.616367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.875386] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6005.452269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.835315] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6009.051275] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6010.715850] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6012.839765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.943889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.946350] Lustre: Skipped 1 previous similar message [ 6014.781163] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6017.080138] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6017.706166] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6028.693198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6030.303642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.306298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.309704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.312349] Lustre: Skipped 2 previous similar messages [ 6032.352233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6036.238916] Lustre: server umount lustre-OST0000 complete [ 6037.472346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.476315] Lustre: Skipped 2 previous similar messages [ 6042.591919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6042.594350] Lustre: Skipped 5 previous similar messages [ 6043.470669] Lustre: server umount lustre-MDT0000 complete [ 6044.609247] LustreError: 406595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790398023 with bad export cookie 527359961368407597 [ 6044.612193] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6044.612440] LustreError: 406595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6044.714795] Lustre: server umount lustre-MDT0001 complete [ 6047.938984] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6048.886428] Key type lgssc unregistered [ 6049.015322] LNet: 410355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6049.017995] LNetError: 410355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6049.029274] LNet: Removed LNI 192.168.202.112@tcp [ 6049.327178] Key type .llcrypt unregistered [ 6049.328840] Key type ._llcrypt unregistered [ 6056.238997] Key type ._llcrypt registered [ 6056.240023] Key type .llcrypt registered [ 6056.273019] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6056.360232] Key type .llcrypt unregistered [ 6056.361191] Key type ._llcrypt unregistered [ 6062.941239] Key type ._llcrypt registered [ 6062.942982] Key type .llcrypt registered [ 6062.982677] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6063.261322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6063.277667] alg: No test for adler32 (adler32-zlib) [ 6064.136190] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 6064.221818] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6065.807168] Key type lgssc registered [ 6066.155060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6068.571710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6068.577243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.705925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.074731] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6073.787989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6073.894472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6075.694159] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6077.863871] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6078.945120] LustreError: 412535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.950030] LustreError: 412535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6079.947631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.968951] LustreError: 412534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.972491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6079.974825] LustreError: 412534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6080.082258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6081.366854] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6090.209079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.210136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.211073] LustreError: Skipped 1 previous similar message [ 6090.215310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.218088] Lustre: Skipped 1 previous similar message [ 6094.671919] Lustre: server umount lustre-OST0000 complete [ 6098.912977] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6098.914767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.917242] Lustre: Skipped 1 previous similar message [ 6098.918442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.448290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.448539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.970677] Lustre: server umount lustre-MDT0000 complete [ 6103.131142] LustreError: 412516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790398082 with bad export cookie 14473558409324886763 [ 6103.134700] LustreError: 412516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6103.134876] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6103.238283] Lustre: server umount lustre-MDT0001 complete [ 6106.786151] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6107.801941] Key type lgssc unregistered [ 6107.928388] LNet: 415680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.931032] LNetError: 415680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.939267] LNet: Removed LNI 192.168.202.112@tcp [ 6108.231852] Key type .llcrypt unregistered [ 6108.233237] Key type ._llcrypt unregistered [ 6117.125162] Key type ._llcrypt registered [ 6117.126629] Key type .llcrypt registered [ 6117.163074] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6117.468185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6117.486679] alg: No test for adler32 (adler32-zlib) [ 6118.341926] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 6118.428735] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6120.007093] Key type lgssc registered [ 6120.379824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6122.947827] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6122.954706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.399404] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6143.519209] LustreError: 417042:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.112@tcp: failed processing log, type 4: rc = -110 [ 6170.143135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.862205] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6174.044758] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6176.295034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.472709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6176.498359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6177.735234] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6180.502135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.621722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6181.855414] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6184.053746] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6185.690967] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6190.049844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6190.051054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6190.052980] Lustre: Skipped 1 previous similar message [ 6193.929859] Lustre: server umount lustre-OST0000 complete [ 6195.168223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.168780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6195.171284] Lustre: Skipped 1 previous similar message [ 6195.172806] Lustre: Skipped 1 previous similar message [ 6200.287868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.289468] Lustre: Skipped 2 previous similar messages [ 6201.174654] Lustre: server umount lustre-MDT0000 complete [ 6202.410189] LustreError: 417050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790398181 with bad export cookie 14826216404616636998 [ 6202.414045] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6202.414668] LustreError: 417050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6202.530517] Lustre: server umount lustre-MDT0001 complete [ 6206.132687] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6207.198104] Key type lgssc unregistered [ 6207.320324] LNet: 420612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6207.322491] LNetError: 420612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6207.331276] LNet: Removed LNI 192.168.202.112@tcp [ 6207.656460] Key type .llcrypt unregistered [ 6207.657559] Key type ._llcrypt unregistered [ 6216.159879] Key type ._llcrypt registered [ 6216.161290] Key type .llcrypt registered [ 6216.198850] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6216.572228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6216.591737] alg: No test for adler32 (adler32-zlib) [ 6217.436831] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 6217.520945] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6219.103148] Key type lgssc registered [ 6219.419167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.754602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.762831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.912494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6224.186517] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6226.811801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6226.938403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6228.346536] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6230.776079] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6232.623375] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6234.903429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6235.028772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6236.973566] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6239.324142] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6248.361908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6252.512336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6252.516365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.522415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.536274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.537670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.930392] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.12@tcp (stopping) [ 6254.932610] Lustre: Skipped 2 previous similar messages [ 6256.859582] Lustre: server umount lustre-MDT0000 complete [ 6257.678434] LustreError: 423844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.12@tcp arrived at 1790398236 with bad export cookie 15887473920969175622 [ 6257.684452] LustreError: 423844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6258.350198] LustreError: 421983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790398237 with bad export cookie 15887473920969174971 [ 6258.354190] LustreError: 421983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6258.354439] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6258.656131] LustreError: 422002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.656765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6258.662387] LustreError: 422002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6258.668694] Lustre: Skipped 1 previous similar message [ 6258.675899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.776214] LustreError: 422728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.776625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.780254] LustreError: 422728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6264.463460] Lustre: server umount lustre-OST0000 complete [ 6266.753528] Lustre: server umount lustre-MDT0001 complete [ 6270.335484] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6271.427935] Key type lgssc unregistered [ 6271.589407] LNet: 425704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.592107] LNetError: 425704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.598466] LNet: Removed LNI 192.168.202.112@tcp [ 6271.944523] Key type .llcrypt unregistered [ 6271.945982] Key type ._llcrypt unregistered [ 6280.658491] Key type ._llcrypt registered [ 6280.659413] Key type .llcrypt registered [ 6280.693766] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6280.963959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6280.971669] alg: No test for adler32 (adler32-zlib) [ 6281.810013] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 6281.889820] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6283.463170] Key type lgssc registered [ 6283.806234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.371783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6286.378146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.526794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.806617] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6291.719614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.854674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.155488] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6295.548478] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6297.346994] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6299.555588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.673498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.541455] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6303.928912] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6311.970429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6317.025545] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.030559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6317.032415] Lustre: Skipped 1 previous similar message [ 6319.446126] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [ 6321.361274] Lustre: server umount lustre-MDT0001 complete [ 6322.144751] LustreError: 427121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.152876] LustreError: 427121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6327.264774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.265273] LustreError: 427102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6327.265616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.265625] Lustre: Skipped 1 previous similar message [ 6327.266119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6329.102591] Lustre: server umount lustre-OST0000 complete [ 6330.428531] Lustre: server umount lustre-MDT0000 complete [ 6334.736782] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6335.712627] Key type lgssc unregistered [ 6335.839452] LNet: 430805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6335.841687] LNetError: 430805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6335.848447] LNet: Removed LNI 192.168.202.112@tcp [ 6336.174543] Key type .llcrypt unregistered [ 6336.175764] Key type ._llcrypt unregistered [ 6344.112069] Key type ._llcrypt registered [ 6344.113544] Key type .llcrypt registered [ 6344.149801] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 6350.163978] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6350.543929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6350.552949] alg: No test for adler32 (adler32-zlib) [ 6351.409298] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 6351.510409] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6353.087195] Key type lgssc registered [ 6353.459937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6365.327720] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6370.373485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6370.379410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.461237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6371.470450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6371.514698] Lustre: lustre-MDT0000: new disk, initializing [ 6371.535654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.542746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6372.934353] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6378.754949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.874851] Lustre: lustre-OST0000: new disk, initializing [ 6378.876435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6378.878094] Lustre: Skipped 1 previous similar message [ 6378.879934] Lustre: 435802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6378.906923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.914042] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6384.554840] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6384.636784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6384.639346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6384.646839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6389.815633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.860540] Lustre: 434787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6389.887881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6389.926820] Lustre: lustre-MDT0001: new disk, initializing [ 6389.970638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6389.984830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6389.988833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6391.330819] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6394.960137] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6395.026914] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6396.370437] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6396.443529] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6398.291036] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.12@tcp (stopping) [ 6403.040655] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6403.041026] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.043470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6403.045942] Lustre: Skipped 1 previous similar message [ 6408.162280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6408.164837] Lustre: Skipped 3 previous similar messages [ 6412.255076] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6412.315362] Lustre: server umount lustre-MDT0001 complete [ 6418.401723] LustreError: 434791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.408521] LustreError: 434791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6423.523640] LustreError: 434796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6423.529035] LustreError: 434796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6425.567990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.570095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.574868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.577157] Lustre: Skipped 1 previous similar message [ 6428.642283] LustreError: 434797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.760928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6433.761317] LustreError: 434796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.763298] Lustre: Skipped 1 previous similar message [ 6438.879162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6438.882825] LustreError: 434797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.941769] Lustre: server umount lustre-OST0000 complete [ 6446.527289] Lustre: server umount lustre-MDT0000 complete [ 6450.966594] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6451.980773] Key type lgssc unregistered [ 6452.116540] LNet: 439546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6452.119823] LNetError: 439546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6452.132526] LNet: Removed LNI 192.168.202.112@tcp [ 6452.483421] Key type .llcrypt unregistered [ 6452.484504] Key type ._llcrypt unregistered [ 6465.488907] Key type ._llcrypt registered [ 6465.490394] Key type .llcrypt registered [ 6465.532384] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 6465.646235] Key type .llcrypt unregistered [ 6465.647702] Key type ._llcrypt unregistered [ 6472.090411] Key type ._llcrypt registered [ 6472.091687] Key type .llcrypt registered [ 6472.130163] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 6472.429750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6472.444352] alg: No test for adler32 (adler32-zlib) [ 6473.301286] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 6473.386349] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 6474.967165] Key type lgssc registered [ 6475.334456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6484.371637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6484.378804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.484755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6485.495427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6485.532715] Lustre: lustre-MDT0000: new disk, initializing [ 6485.556414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.562239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6486.944360] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6491.149783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.196136] Lustre: 442886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6491.222795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6491.224332] Lustre: Skipped 1 previous similar message [ 6491.260875] Lustre: lustre-MDT0001: new disk, initializing [ 6491.300989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6491.315100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6491.318703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6492.687050] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6496.495696] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6498.299804] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6500.547066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.651460] Lustre: lustre-OST0000: new disk, initializing [ 6500.653655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6500.657101] Lustre: 444854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6500.687499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.697524] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6506.333486] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.560413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6509.563423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6509.593091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6514.658251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6514.660036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.662729] Lustre: Skipped 2 previous similar messages [ 6514.667682] Lustre: Skipped 2 previous similar messages [ 6519.776714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6519.778541] Lustre: Skipped 2 previous similar messages [ 6520.151187] Lustre: server umount lustre-MDT0000 complete [ 6521.498903] LustreError: 443682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790398500 with bad export cookie 15223404439307982686 [ 6521.504734] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6521.505682] LustreError: 443682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6521.635551] Lustre: server umount lustre-MDT0001 complete [ 6533.213495] Lustre: server umount lustre-OST0000 complete [ 6536.615493] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:55:15 (1790398515) [ 6539.172067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.348867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.712874] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6543.487916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.612304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6544.919813] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6547.367680] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6549.152244] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6564.450253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6564.522604] Lustre: Found index 0 for lustre-OST0000, updating log [ 6564.580166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.556533] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 6569.136117] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6585.065859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6594.528290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6594.532029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6594.538498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6597.093666] Lustre: server umount lustre-OST0000 complete [ 6609.754392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.755002] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6609.757185] Lustre: Skipped 2 previous similar messages [ 6609.759195] Lustre: Skipped 3 previous similar messages [ 6615.008261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6615.010052] Lustre: Skipped 1 previous similar message [ 6622.879250] LNet: 1 local NIs in recovery (showing 1): 192.168.202.112@tcp [ 6624.223134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6624.283562] Lustre: server umount lustre-MDT0000 complete [ 6625.491171] LustreError: 446864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790398604 with bad export cookie 15223404439307984065 [ 6625.494401] LustreError: 446864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6625.495031] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6625.613375] Lustre: server umount lustre-MDT0001 complete [ 6628.280750] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:56:47 (1790398607) [ 7208.844721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.076140] LustreError: 453319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7209.081689] LustreError: 453319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7209.106349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.628775] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7214.115314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.276717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.832797] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7218.685658] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7220.745155] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.258471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7236.379481] Lustre: Found index 0 for lustre-OST0000, updating log [ 7236.492878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.377731] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7240.731903] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7241.456168] Lustre: server umount lustre-OST0000 complete [ 7245.279873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7245.282878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.287356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7245.289875] Lustre: Skipped 6 previous similar messages [ 7247.840275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.840752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.960171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7252.962090] Lustre: Skipped 2 previous similar messages [ 7257.055213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7257.154903] Lustre: server umount lustre-MDT0000 complete [ 7258.080213] LustreError: 453320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7258.085118] LustreError: 453320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7258.500955] LustreError: 453345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399237 with bad export cookie 15223404439307985136 [ 7258.503662] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7258.507472] LustreError: 453345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7258.637686] Lustre: server umount lustre-MDT0001 complete [ 7262.879275] LNet: 1 local NIs in recovery (showing 1): 192.168.202.112@tcp [ 7276.978405] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7279.927684] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7293.072394] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:07:52 (1790399272) [ 7298.845465] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7299.870189] Key type lgssc unregistered [ 7299.995357] LNet: 459964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7299.997515] LNetError: 459964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7366.565427] LNet: Removed LNI 192.168.202.112@tcp [ 7366.950514] Key type .llcrypt unregistered [ 7366.952441] Key type ._llcrypt unregistered [ 7373.707053] Key type ._llcrypt registered [ 7373.708771] Key type .llcrypt registered [ 7373.751433] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7374.098872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7374.144097] alg: No test for adler32 (adler32-zlib) [ 7375.013697] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 7375.108551] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7376.687195] Key type lgssc registered [ 7377.163886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7378.590560] LNet: Added LNI 192.168.202.112@tcp100 [8/256/0/180] [ 7378.881078] LNet: Added LNI 192.168.202.112@tcp101 [8/256/0/180] [ 7379.180811] LNet: Added LNI 192.168.202.112@tcp102 [8/256/0/180] [ 7379.562127] LNet: Added LNI 192.168.202.112@tcp103 [8/256/0/180] [ 7379.948262] LNet: Added LNI 192.168.202.112@tcp104 [8/256/0/180] [ 7380.404970] LNet: Added LNI 192.168.202.112@tcp105 [8/256/0/180] [ 7380.807290] LNet: Added LNI 192.168.202.112@tcp106 [8/256/0/180] [ 7381.235507] LNet: Added LNI 192.168.202.112@tcp107 [8/256/0/180] [ 7381.669713] LNet: Added LNI 192.168.202.112@tcp108 [8/256/0/180] [ 7382.089075] LNet: Added LNI 192.168.202.112@tcp109 [8/256/0/180] [ 7382.525967] LNet: Added LNI 192.168.202.112@tcp110 [8/256/0/180] [ 7382.925984] LNet: Added LNI 192.168.202.112@tcp111 [8/256/0/180] [ 7383.269198] LNet: Added LNI 192.168.202.112@tcp112 [8/256/0/180] [ 7383.547114] LNet: Added LNI 192.168.202.112@tcp113 [8/256/0/180] [ 7383.815967] LNet: Added LNI 192.168.202.112@tcp114 [8/256/0/180] [ 7384.138343] LNet: Added LNI 192.168.202.112@tcp115 [8/256/0/180] [ 7384.513948] LNet: Added LNI 192.168.202.112@tcp116 [8/256/0/180] [ 7384.922834] LNet: Added LNI 192.168.202.112@tcp117 [8/256/0/180] [ 7385.370365] LNet: Added LNI 192.168.202.112@tcp118 [8/256/0/180] [ 7385.823797] LNet: Added LNI 192.168.202.112@tcp119 [8/256/0/180] [ 7386.151559] LNet: Added LNI 192.168.202.112@tcp120 [8/256/0/180] [ 7386.496908] LNet: Added LNI 192.168.202.112@tcp121 [8/256/0/180] [ 7386.871464] LNet: Added LNI 192.168.202.112@tcp122 [8/256/0/180] [ 7387.181762] LNet: Added LNI 192.168.202.112@tcp123 [8/256/0/180] [ 7387.469168] LNet: Added LNI 192.168.202.112@tcp124 [8/256/0/180] [ 7387.755333] LNet: Added LNI 192.168.202.112@tcp125 [8/256/0/180] [ 7388.037302] LNet: Added LNI 192.168.202.112@tcp126 [8/256/0/180] [ 7388.319485] LNet: Added LNI 192.168.202.112@tcp127 [8/256/0/180] [ 7388.603846] LNet: Added LNI 192.168.202.112@tcp128 [8/256/0/180] [ 7388.887419] LNet: Added LNI 192.168.202.112@tcp129 [8/256/0/180] [ 7389.180099] LNet: Added LNI 192.168.202.112@tcp130 [8/256/0/180] [ 7389.469512] LNet: Added LNI 192.168.202.112@tcp131 [8/256/0/180] [ 7389.751692] LNet: Added LNI 192.168.202.112@tcp132 [8/256/0/180] [ 7390.030556] LNet: Added LNI 192.168.202.112@tcp133 [8/256/0/180] [ 7390.304447] LNet: Added LNI 192.168.202.112@tcp134 [8/256/0/180] [ 7390.597440] LNet: Added LNI 192.168.202.112@tcp135 [8/256/0/180] [ 7392.672447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7392.677960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.714360] Lustre: 462956:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.112@tcp: can't add NID 192.168.202.112@tcp130: rc = -75 [ 7393.739879] LNetError: 460420:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.112 [ 7393.754944] LNetError: Protocol error connecting to 192.168.202.112@tcp118 at host 192.168.202.112:988: is it running a compatible version of Lustre? [ 7393.758070] LNetError: Skipped 1 previous similar message [ 7393.807538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7393.816952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7393.857301] Lustre: lustre-MDT0000: new disk, initializing [ 7393.877821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7393.883796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7394.591154] LNet: 1 local NIs in recovery (showing 1): 192.168.202.112@tcp128 [ 7394.592136] LNetError: 462980:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.112 [ 7394.595414] LNetError: 462980:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 7394.597521] LNetError: Protocol error connecting to 192.168.202.112@tcp at host 192.168.202.112:988: is it running a compatible version of Lustre? [ 7394.600240] LNetError: Skipped 5 previous similar messages [ 7395.172413] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7396.640175] LNetError: 462980:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.112 [ 7396.643055] LNetError: 462980:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 7396.645400] LNetError: Protocol error connecting to 192.168.202.112@tcp128 at host 192.168.202.112:988: is it running a compatible version of Lustre? [ 7396.648689] LNetError: Skipped 5 previous similar messages [ 7399.139129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7399.189823] Lustre: 462973:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7399.215548] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7399.217204] Lustre: Skipped 1 previous similar message [ 7399.252851] Lustre: lustre-MDT0001: new disk, initializing [ 7399.264852] Lustre: 463776:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.112@tcp: can't add NID 192.168.202.112@tcp130: rc = -75 [ 7399.267308] Lustre: 463776:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7399.308109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7399.319783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7399.322741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7400.628249] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7400.736204] LNetError: 460421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.112 [ 7400.739585] LNetError: Protocol error connecting to 192.168.202.112@tcp128 at host 192.168.202.112:988: is it running a compatible version of Lustre? [ 7402.293466] Lustre: 464156:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.112@tcp: can't add NID 192.168.202.112@tcp130: rc = -75 [ 7402.296573] Lustre: 464156:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7404.179645] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7405.933685] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7408.084344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.209484] Lustre: lustre-OST0000: new disk, initializing [ 7408.211490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7408.215237] Lustre: 464949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7408.249812] Lustre: 464954:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.112@tcp: can't add NID 192.168.202.112@tcp130: rc = -75 [ 7408.252586] Lustre: 464954:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7408.268955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7408.927939] LNetError: 462976:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.112 [ 7408.930572] LNetError: Protocol error connecting to 192.168.202.112@tcp128 at host 192.168.202.112:988: is it running a compatible version of Lustre? [ 7410.117736] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7413.789480] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7416.028615] Lustre: server umount lustre-OST0000 complete [ 7418.354430] Lustre: 466090:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.112@tcp: can't add NID 192.168.202.112@tcp130: rc = -75 [ 7418.358850] Lustre: 466090:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7418.370387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.371170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.375419] Lustre: Skipped 1 previous similar message [ 7423.317548] Lustre: server umount lustre-MDT0000 complete [ 7424.287873] LNetError: 462977:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.112 [ 7424.292723] LNetError: Protocol error connecting to 192.168.202.112@tcp118 at host 192.168.202.112:988: is it running a compatible version of Lustre? [ 7424.500273] LustreError: 463019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399403 with bad export cookie 721464187454039826 [ 7424.503325] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7424.503360] LustreError: 463019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7424.617272] Lustre: server umount lustre-MDT0001 complete [ 7428.005377] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7428.954674] Key type lgssc unregistered [ 7429.095033] LNet: 466848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7429.097704] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7429.099678] LNet: Removed LNI 192.168.202.112@tcp [ 7430.111129] LNet: Removed LNI 192.168.202.112@tcp100 [ 7430.112530] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7430.114366] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7430.116329] LNet: Removed LNI 192.168.202.112@tcp101 [ 7431.135131] LNet: Removed LNI 192.168.202.112@tcp102 [ 7431.136289] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7431.138172] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7431.140395] LNet: Removed LNI 192.168.202.112@tcp103 [ 7432.159129] LNet: Removed LNI 192.168.202.112@tcp104 [ 7433.183157] LNet: Removed LNI 192.168.202.112@tcp105 [ 7433.184833] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7433.187072] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7433.190074] LNet: Removed LNI 192.168.202.112@tcp106 [ 7434.207185] LNet: Removed LNI 192.168.202.112@tcp107 [ 7435.231109] LNet: Removed LNI 192.168.202.112@tcp108 [ 7436.255117] LNet: Removed LNI 192.168.202.112@tcp109 [ 7437.279174] LNet: Removed LNI 192.168.202.112@tcp110 [ 7437.281334] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.283874] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7437.285916] LNet: Removed LNI 192.168.202.112@tcp111 [ 7438.303160] LNet: Removed LNI 192.168.202.112@tcp112 [ 7439.327168] LNet: Removed LNI 192.168.202.112@tcp113 [ 7440.351133] LNet: Removed LNI 192.168.202.112@tcp114 [ 7441.375102] LNet: Removed LNI 192.168.202.112@tcp115 [ 7442.399185] LNet: Removed LNI 192.168.202.112@tcp116 [ 7443.423157] LNet: Removed LNI 192.168.202.112@tcp117 [ 7444.447153] LNet: Removed LNI 192.168.202.112@tcp118 [ 7445.471107] LNet: Removed LNI 192.168.202.112@tcp119 [ 7445.472453] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7445.474381] LNetError: 466848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7445.476476] LNet: Removed LNI 192.168.202.112@tcp120 [ 7446.495180] LNet: Removed LNI 192.168.202.112@tcp121 [ 7447.519146] LNet: Removed LNI 192.168.202.112@tcp122 [ 7448.543163] LNet: Removed LNI 192.168.202.112@tcp123 [ 7449.567116] LNet: Removed LNI 192.168.202.112@tcp124 [ 7450.591122] LNet: Removed LNI 192.168.202.112@tcp125 [ 7451.615143] LNet: Removed LNI 192.168.202.112@tcp126 [ 7452.639181] LNet: Removed LNI 192.168.202.112@tcp127 [ 7453.663149] LNet: Removed LNI 192.168.202.112@tcp128 [ 7454.687105] LNet: Removed LNI 192.168.202.112@tcp129 [ 7454.688479] LNet: Removed LNI 192.168.202.112@tcp130 [ 7454.689551] LNet: Removed LNI 192.168.202.112@tcp131 [ 7454.690585] LNet: Removed LNI 192.168.202.112@tcp132 [ 7454.691682] LNet: Removed LNI 192.168.202.112@tcp133 [ 7454.692756] LNet: Removed LNI 192.168.202.112@tcp134 [ 7454.699505] LNet: Removed LNI 192.168.202.112@tcp135 [ 7454.984070] Key type .llcrypt unregistered [ 7454.986086] Key type ._llcrypt unregistered [ 7462.375484] Key type ._llcrypt registered [ 7462.376391] Key type .llcrypt registered [ 7462.410749] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7468.395213] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7468.726035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7468.771079] alg: No test for adler32 (adler32-zlib) [ 7469.615085] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 7469.696222] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7471.271138] Key type lgssc registered [ 7471.627418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7482.433223] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:11:01 (1790399461) [ 7488.699449] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7489.769697] Key type lgssc unregistered [ 7489.900401] LNet: 470905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7489.902593] LNetError: 470905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7489.915426] LNet: Removed LNI 192.168.202.112@tcp [ 7490.211303] Key type .llcrypt unregistered [ 7490.212261] Key type ._llcrypt unregistered [ 7496.659690] Key type ._llcrypt registered [ 7496.661067] Key type .llcrypt registered [ 7496.699009] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7497.014761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7497.020481] alg: No test for adler32 (adler32-zlib) [ 7497.863687] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 7497.951089] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7499.527161] Key type lgssc registered [ 7499.871242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7502.705379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7502.711158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7503.797376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7503.806726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7503.844630] Lustre: lustre-MDT0000: new disk, initializing [ 7503.865541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7503.871016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7505.239358] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7509.329951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.380992] Lustre: 472089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7509.410493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7509.412177] Lustre: Skipped 1 previous similar message [ 7509.448949] Lustre: lustre-MDT0001: new disk, initializing [ 7509.481932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7509.501261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7509.504241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7510.832760] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7514.612063] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7516.560559] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7518.906594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7519.025827] Lustre: lustre-OST0000: new disk, initializing [ 7519.028201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7519.031752] Lustre: 474061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7519.061089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.868388] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7524.504200] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7525.672587] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing lnet_if_list [ 7526.913371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7526.916870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7526.972137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7527.387532] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7527.398316] LNet: Added LNI 192.168.202.112@tcp10 [8/256/0/180] [ 7529.267412] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7529.278085] LNet: Added LNI 192.168.202.112@tcp11 [8/256/0/180] [ 7530.604130] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7530.614835] LNet: Added LNI 192.168.202.112@tcp12 [8/256/0/180] [ 7532.114226] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7532.127671] LNet: Added LNI 192.168.202.112@tcp13 [8/256/0/180] [ 7533.586089] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7533.598084] LNet: Added LNI 192.168.202.112@tcp14 [8/256/0/180] [ 7534.982391] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7534.993229] LNet: Added LNI 192.168.202.112@tcp15 [8/256/0/180] [ 7536.623346] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7536.638703] LNet: Added LNI 192.168.202.112@tcp16 [8/256/0/180] [ 7538.500532] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7538.518369] LNet: Added LNI 192.168.202.112@tcp17 [8/256/0/180] [ 7540.312750] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7540.323942] LNet: Added LNI 192.168.202.112@tcp18 [8/256/0/180] [ 7542.084056] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7542.105928] LNet: Added LNI 192.168.202.112@tcp19 [8/256/0/180] [ 7543.919278] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7543.930558] LNet: Added LNI 192.168.202.112@tcp20 [8/256/0/180] [ 7545.757510] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7545.770463] LNet: Added LNI 192.168.202.112@tcp21 [8/256/0/180] [ 7547.666094] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7547.681031] LNet: Added LNI 192.168.202.112@tcp22 [8/256/0/180] [ 7549.645040] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7549.656682] LNet: Added LNI 192.168.202.112@tcp23 [8/256/0/180] [ 7551.674911] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7551.692738] LNet: Added LNI 192.168.202.112@tcp24 [8/256/0/180] [ 7553.761592] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7553.772109] LNet: Added LNI 192.168.202.112@tcp25 [8/256/0/180] [ 7555.989834] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7556.010818] LNet: Added LNI 192.168.202.112@tcp26 [8/256/0/180] [ 7558.319488] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7558.336819] LNet: Added LNI 192.168.202.112@tcp27 [8/256/0/180] [ 7560.664068] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7560.680395] LNet: Added LNI 192.168.202.112@tcp28 [8/256/0/180] [ 7563.188510] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7563.199983] LNet: Added LNI 192.168.202.112@tcp29 [8/256/0/180] [ 7565.721552] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7565.741829] LNet: Added LNI 192.168.202.112@tcp30 [8/256/0/180] [ 7568.397061] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7568.412531] LNet: Added LNI 192.168.202.112@tcp31 [8/256/0/180] [ 7570.767047] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7570.782309] LNet: Added LNI 192.168.202.112@tcp32 [8/256/0/180] [ 7573.134424] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7573.145590] LNet: Added LNI 192.168.202.112@tcp33 [8/256/0/180] [ 7575.414280] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7575.425414] LNet: Added LNI 192.168.202.112@tcp34 [8/256/0/180] [ 7577.781597] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7577.792110] LNet: Added LNI 192.168.202.112@tcp35 [8/256/0/180] [ 7580.232419] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7580.244271] LNet: Added LNI 192.168.202.112@tcp36 [8/256/0/180] [ 7582.539892] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7582.550404] LNet: Added LNI 192.168.202.112@tcp37 [8/256/0/180] [ 7584.758480] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7584.769853] LNet: Added LNI 192.168.202.112@tcp38 [8/256/0/180] [ 7587.311398] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7587.321749] LNet: Added LNI 192.168.202.112@tcp39 [8/256/0/180] [ 7589.788941] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7589.800849] LNet: Added LNI 192.168.202.112@tcp40 [8/256/0/180] [ 7592.410268] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7592.426601] LNet: Added LNI 192.168.202.112@tcp41 [8/256/0/180] [ 7595.061756] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7595.074293] LNet: Added LNI 192.168.202.112@tcp42 [8/256/0/180] [ 7597.752803] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7597.766995] LNet: Added LNI 192.168.202.112@tcp43 [8/256/0/180] [ 7600.420897] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7600.432882] LNet: Added LNI 192.168.202.112@tcp44 [8/256/0/180] [ 7603.120904] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7603.135243] LNet: Added LNI 192.168.202.112@tcp45 [8/256/0/180] [ 7605.863257] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7605.874630] LNet: Added LNI 192.168.202.112@tcp46 [8/256/0/180] [ 7608.562724] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7608.573590] LNet: Added LNI 192.168.202.112@tcp47 [8/256/0/180] [ 7611.225113] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7611.235530] LNet: Added LNI 192.168.202.112@tcp48 [8/256/0/180] [ 7614.161660] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7614.172618] LNet: Added LNI 192.168.202.112@tcp49 [8/256/0/180] [ 7617.054681] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7617.068662] LNet: Added LNI 192.168.202.112@tcp50 [8/256/0/180] [ 7627.234516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7627.236926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.240133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.244844] Lustre: Skipped 2 previous similar messages [ 7627.249899] Lustre: Skipped 2 previous similar messages [ 7630.893262] Lustre: server umount lustre-MDT0000 complete [ 7632.240544] LustreError: 472126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399611 with bad export cookie 11388415948296429173 [ 7632.243132] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7632.243471] LustreError: 472126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7632.364185] Lustre: server umount lustre-MDT0001 complete [ 7640.158593] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7643.282806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.735255] Lustre: 471362:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790399611/real 1790399611] req@00000000b0e4e1f7 x1877369922082048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790399627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7648.751493] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7652.831071] Lustre: 471364:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790399616/real 1790399616] req@0000000038a6e67d x1877369922082176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790399632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7656.991639] LustreError: 471361:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000038a6e67d x1877369922082944/t0(0) o250->MGC192.168.202.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7657.166719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7657.951090] Lustre: 471365:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790399621/real 1790399621] req@000000002ea0336a x1877369922082304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790399637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7658.585081] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7661.979111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.173619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.071173] Lustre: 471363:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790399626/real 1790399626] req@000000008c0e8910 x1877369922082560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790399642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7663.710102] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7666.923225] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7667.173845] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7668.197588] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7669.132596] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7677.408985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7677.411757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.415516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7678.436445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7678.438105] Lustre: Skipped 2 previous similar messages [ 7682.643672] Lustre: server umount lustre-OST0000 complete [ 7687.648713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7687.654097] LustreError: Skipped 1 previous similar message [ 7687.657067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7687.665240] Lustre: Skipped 1 previous similar message [ 7687.668969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.792991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7693.795431] Lustre: Skipped 3 previous similar messages [ 7698.399212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7698.484305] Lustre: server umount lustre-MDT0000 complete [ 7698.912745] LustreError: 498249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.918063] LustreError: 498249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7700.079819] LustreError: 472084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399679 with bad export cookie 11388415948296431210 [ 7700.085520] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7700.232949] Lustre: server umount lustre-MDT0001 complete [ 7704.570220] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7705.579708] Key type lgssc unregistered [ 7705.728249] LNet: 501022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7705.732106] LNetError: 501022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7705.735263] LNet: Removed LNI 192.168.202.112@tcp [ 7705.736354] LNet: Removed LNI 192.168.202.112@tcp10 [ 7705.737724] LNet: Removed LNI 192.168.202.112@tcp11 [ 7705.739064] LNet: Removed LNI 192.168.202.112@tcp12 [ 7705.740978] LNet: Removed LNI 192.168.202.112@tcp13 [ 7705.743375] LNet: Removed LNI 192.168.202.112@tcp14 [ 7705.744675] LNet: Removed LNI 192.168.202.112@tcp15 [ 7705.745994] LNet: Removed LNI 192.168.202.112@tcp16 [ 7705.747244] LNet: Removed LNI 192.168.202.112@tcp17 [ 7705.748414] LNet: Removed LNI 192.168.202.112@tcp18 [ 7705.749783] LNet: Removed LNI 192.168.202.112@tcp19 [ 7705.750965] LNet: Removed LNI 192.168.202.112@tcp20 [ 7705.752121] LNet: Removed LNI 192.168.202.112@tcp21 [ 7705.753308] LNet: Removed LNI 192.168.202.112@tcp22 [ 7705.754387] LNet: Removed LNI 192.168.202.112@tcp23 [ 7705.755429] LNet: Removed LNI 192.168.202.112@tcp24 [ 7705.756575] LNet: Removed LNI 192.168.202.112@tcp25 [ 7705.758062] LNet: Removed LNI 192.168.202.112@tcp26 [ 7705.760455] LNet: Removed LNI 192.168.202.112@tcp27 [ 7705.762095] LNet: Removed LNI 192.168.202.112@tcp28 [ 7705.763207] LNet: Removed LNI 192.168.202.112@tcp29 [ 7705.764329] LNet: Removed LNI 192.168.202.112@tcp30 [ 7705.765890] LNet: Removed LNI 192.168.202.112@tcp31 [ 7705.768074] LNet: Removed LNI 192.168.202.112@tcp32 [ 7705.769712] LNet: Removed LNI 192.168.202.112@tcp33 [ 7705.770719] LNet: Removed LNI 192.168.202.112@tcp34 [ 7705.771779] LNet: Removed LNI 192.168.202.112@tcp35 [ 7705.772846] LNet: Removed LNI 192.168.202.112@tcp36 [ 7705.773977] LNet: Removed LNI 192.168.202.112@tcp37 [ 7705.775118] LNet: Removed LNI 192.168.202.112@tcp38 [ 7705.776195] LNet: Removed LNI 192.168.202.112@tcp39 [ 7705.777241] LNet: Removed LNI 192.168.202.112@tcp40 [ 7705.778321] LNet: Removed LNI 192.168.202.112@tcp41 [ 7705.779381] LNet: Removed LNI 192.168.202.112@tcp42 [ 7705.781109] LNet: Removed LNI 192.168.202.112@tcp43 [ 7705.782454] LNet: Removed LNI 192.168.202.112@tcp44 [ 7705.784306] LNet: Removed LNI 192.168.202.112@tcp45 [ 7705.785331] LNet: Removed LNI 192.168.202.112@tcp46 [ 7705.786353] LNet: Removed LNI 192.168.202.112@tcp47 [ 7705.787849] LNet: Removed LNI 192.168.202.112@tcp48 [ 7705.789678] LNet: Removed LNI 192.168.202.112@tcp49 [ 7705.797290] LNet: Removed LNI 192.168.202.112@tcp50 [ 7706.115107] Key type .llcrypt unregistered [ 7706.116390] Key type ._llcrypt unregistered [ 7712.561610] Key type ._llcrypt registered [ 7712.563180] Key type .llcrypt registered [ 7712.600942] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7718.528954] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7718.877719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7718.888479] alg: No test for adler32 (adler32-zlib) [ 7719.760275] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 7719.850745] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7721.431185] Key type lgssc registered [ 7721.968686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7734.561439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7734.568058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.661963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7735.673562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7735.710700] Lustre: lustre-MDT0000: new disk, initializing [ 7735.733941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.740681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7737.098554] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7741.570749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.625862] Lustre: 504187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7741.659758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7741.662264] Lustre: Skipped 1 previous similar message [ 7741.707223] Lustre: lustre-MDT0001: new disk, initializing [ 7741.741153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7741.753331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7741.756426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7743.239255] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7747.720487] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7749.661729] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7753.241570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7753.352720] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7753.397447] Lustre: lustre-OST0000: new disk, initializing [ 7753.400816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7753.406653] Lustre: 506204:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7753.458179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7753.520730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7753.523635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7753.533975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 7755.854759] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7760.467867] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7763.941217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7763.941633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.943422] LustreError: Skipped 1 previous similar message [ 7763.947194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7763.949147] Lustre: Skipped 1 previous similar message [ 7765.744157] Lustre: server umount lustre-OST0000 complete [ 7769.057653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.058294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.064108] Lustre: Skipped 2 previous similar messages [ 7773.775395] Lustre: server umount lustre-MDT0000 complete [ 7775.519747] LustreError: 504178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399754 with bad export cookie 2900227170990090732 [ 7775.526232] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.529298] LustreError: 504178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7775.677351] Lustre: server umount lustre-MDT0001 complete [ 7780.603784] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7782.022894] Key type lgssc unregistered [ 7782.189621] LNet: 508107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7782.193488] LNetError: 508107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7782.204510] LNet: Removed LNI 192.168.202.112@tcp [ 7782.660591] Key type .llcrypt unregistered [ 7782.663051] Key type ._llcrypt unregistered [ 7791.526484] Key type ._llcrypt registered [ 7791.528224] Key type .llcrypt registered [ 7791.584644] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7791.902705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7791.920443] alg: No test for adler32 (adler32-zlib) [ 7792.797884] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 7792.905352] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7794.487190] Key type lgssc registered [ 7795.010231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7798.278430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7798.287235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.476762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7801.005869] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7805.178853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7805.403569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7807.096465] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7810.811622] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7813.055541] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7816.371792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.589500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.155674] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7821.967259] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7837.460410] loop0: detected capacity change from 0 to 204800000 [ 7837.524672] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7837.528940] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7837.533665] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7840.416466] loop0: detected capacity change from 0 to 204800000 [ 7840.539220] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7840.542977] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7840.548051] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7842.882087] loop0: detected capacity change from 0 to 204800000 [ 7845.261829] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7845.314229] Lustre: fs15246-OST0000: new disk, initializing [ 7845.315932] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7845.318235] Lustre: 512964:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7845.347161] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7848.140233] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7853.131795] loop1: detected capacity change from 0 to 204800000 [ 7855.542953] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.593322] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7855.620427] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7855.659335] Lustre: fs15246-MDT0000: new disk, initializing [ 7855.728840] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.742868] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7855.807082] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7856.864468] Lustre: 514088:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000747978c3 x1877370231360128/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1790399852 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7857.847579] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7865.954702] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7865.960406] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7865.969952] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7866.245319] Lustre: server umount fs15246-MDT0000 complete [ 7869.419171] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7869.429228] LustreError: 509345:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.438554] LustreError: 509345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7872.573200] Lustre: server umount fs15246-OST0000 complete [ 7879.650096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.651191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7879.653736] Lustre: Skipped 1 previous similar message [ 7882.831616] Lustre: server umount lustre-OST0000 complete [ 7884.768425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7884.769068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7884.772611] Lustre: Skipped 1 previous similar message [ 7884.774468] Lustre: Skipped 1 previous similar message [ 7889.889725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7889.893667] Lustre: Skipped 2 previous similar messages [ 7890.226802] Lustre: server umount lustre-MDT0000 complete [ 7891.746432] LustreError: 516141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399871 with bad export cookie 9183132849316990860 [ 7891.749303] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7891.753949] LustreError: 516141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7891.904694] Lustre: server umount lustre-MDT0001 complete [ 7895.892506] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 7897.056328] Key type lgssc unregistered [ 7897.196473] LNet: 517180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7897.199451] LNetError: 517180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7897.209645] LNet: Removed LNI 192.168.202.112@tcp [ 7897.689606] Key type .llcrypt unregistered [ 7897.691294] Key type ._llcrypt unregistered [ 7905.559038] Key type ._llcrypt registered [ 7905.560200] Key type .llcrypt registered [ 7905.599650] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7905.968970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7905.987958] alg: No test for adler32 (adler32-zlib) [ 7906.841510] Lustre: Lustre: Build Version: 2.17.58_4_g2dccaf1 [ 7906.930867] LNet: Added LNI 192.168.202.112@tcp [8/256/0/180] [ 7908.519138] Key type lgssc registered [ 7908.930092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7921.116800] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7924.345180] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 7935.691195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7935.697565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7936.791520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7936.803074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7936.840657] Lustre: lustre-MDT0000: new disk, initializing [ 7936.861076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.866457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7938.201079] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7942.940473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.002740] Lustre: 521610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7943.028240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7943.030361] Lustre: Skipped 1 previous similar message [ 7943.074655] Lustre: lustre-MDT0001: new disk, initializing [ 7943.112351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7943.128211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7943.132419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7944.527188] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7948.112093] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7949.992736] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7952.173810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7952.280623] Lustre: lustre-OST0000: new disk, initializing [ 7952.282816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7952.286469] Lustre: 523577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7952.318549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.691187] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_default_debug -1 all [ 7958.854530] Lustre: DEBUG MARKER: oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7960.569358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7960.572804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7960.609873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7960.636875] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7960.708068] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7962.141580] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7962.209828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7965.665320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7965.667865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.668655] LustreError: Skipped 1 previous similar message [ 7965.673406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7965.678479] Lustre: Skipped 1 previous similar message [ 7969.008992] Lustre: server umount lustre-OST0000 complete [ 7970.784918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.785623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.792666] Lustre: Skipped 2 previous similar messages [ 7975.905157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.910439] Lustre: Skipped 2 previous similar messages [ 7976.854952] Lustre: server umount lustre-MDT0000 complete [ 7978.547230] LustreError: 521601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790399957 with bad export cookie 17209864696791401223 [ 7978.556193] LustreError: 521601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7978.556849] LustreError: MGC192.168.202.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7978.695094] Lustre: server umount lustre-MDT0001 complete [ 7987.914802] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing set_hostid [ 7991.671145] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing load_modules_local [ 8003.686148] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7702 sec ========== 01:19:42 (1790399982) [ 8004.217233] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:19:43 (1790399983) === [ 8005.365674] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:19:44 (1790399984) === [ 8021.864451] Lustre: DEBUG MARKER: oleg212-server.virtnet: executing unload_modules_local [ 8023.439850] Key type lgssc unregistered [ 8023.638683] LNet: 530209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8023.643920] LNetError: 530209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8023.658614] LNet: Removed LNI 192.168.202.112@tcp [ 8024.202854] Key type .llcrypt unregistered [ 8024.205102] Key type ._llcrypt unregistered